The Weather Company, an IBM Business, talks about the analytics tools they use to help energy companies predict electricity consumption and traders understand expected changes in the weather. They also talk about how load forecasting is rapidly becoming all about handling Big Data, and how it’s going way beyond the spreadsheet systems of yesterday.
I caught up with Ed Cuoco, director, Data Science and Analytics at The Weather Company.
First, what’s the current status of advanced new data analytics tools and data scientists to improve forecasts for electricity consumption?
Cuoco said: “The sophistication and refinement in demand forecasting has increased by leaps and bounds over the last 5 years. The near-real time forward demand curve has become omnipresent within the demand, generation and merchant worlds. Today, the value of forward analytics lies in improving marginal accuracy and timeliness as well as including more diverse and volatile data sources (weather, renewable generation, etc) and applying the insight into more complex business problems where increased speed, accuracy and precision become ever more important.
“Indeed, forecasts have moved beyond capital planning, pricing auctions and energy to become critical across the energy infrastructure, from portfolio management and risk mitigation through demand management and even near-real time operational efficiency.”
So, why is there so much interest in data scientists?
He added that load data and related data sets are seen as core business assets; extracting the value of this data has become a critical task across energy-related businesses. This increased use also increases the criticality of contextual data quality (i.e., is load data that was good enough for thinking about yearly capital spend also good enough for near-real time trading?).
The role of the data scientist lies in the nexus of these two tasks; unlocking more value from diverse data sets (load data, customer data, operational data) while also helping utilities, ISOs, traders and other stake holders understand and overcome limitations in the data themselves to continue to drive more refined understanding and action.
Next, we need to know what sort of analytics tools does The Weather Company work on/with, and what can those do?
Cuoco said: “We use most of the common analytical tools in the market, but what differentiates us is our underlying data platform and its ability to handle huge amounts of weather and load data. Every day The Weather Company maps 62 vertical miles of the atmosphere, all around the globe, to deliver 35+ billion forecasts along with 100+ terabytes of data.
“Four years ago, The Weather Company went through a transformation to a new cloud-based, cloud-agnostic, data-driven infrastructure. That “SUN” platform is an efficient Data as a Service (DaaS) platform that reliably handles these incredible workloads, leveraging 249 different open source tools with proprietary capabilities.
“Most of the platform was written in Scala, and a few of the technologies it leverages include Cassandra, Spark, Riak, and Redis. This platform allows us to turn Big Data into better decision making and provide demand forecasts to energy traders and utilities.
Finally, how is load forecasting now becoming all about handling Big Data?
The robustness, availability and quality of data sets varies tremendously from firm to firm and geography to geography, so there’s a lack of standardization and maturity. In some cases, load data is flowing in near real time at the meter level, in others, only at the grid level, and some in between.
With the implication that all the analytics are the same in principal, the real value to the industry is unlocking the value in the data sets as they exist now, not simply saying “when everything is robust, pristine and perfect, then you’ll see the value.”
What questions can be answered and how confident can one be in that answer is the crux of the big data question in this space. At the same time, load forecasting plays a central role across the energy and utility space, helping drive everything — from real-time demand response to optimizing the use of existing grid infrastructure and preventing catastrophic failures.
These predictions look minutes ahead for traders and days ahead for an ISO or utility, and the business needs to switch seamlessly between geographic and temporal filters. As the questions become more complex, interval meter data is critical but insufficient. Complex weather data, sensor data and customer behavior data all play a role in truly forecasting load and then applying that forecast to finance, operations and customer service.
And, how is it going beyond the spreadsheet systems of yesterday?
Cuoco concluded: “As analytics begin driving ongoing operations as well as strategic planning, the need to handle volume, speed and complexity dwarfs what even the most complex set of spreadsheets can accomplish while also pushing the limits on the underlying products used to create them.
“Further, when this increased business relevance is combined with the more robust, complex, and diverse data, the constant need to revisit and update even the best models becomes critical; analysis is no longer sufficient and, even now, we’re seeing the combination of advanced statistics and machine learning evolve.
“Soon, companies will need solutions that truly learn, interact and reason. Spreadsheets can’t keep up with the load or the complexity and, when pushed to their limits, increase the chances of critical failures in key analytical processes.”