The Weather Company, an IBM Business, talks about the analytics tools they use to help energy companies predict electricity consumption and traders understand expected changes in the weather. They also talk about how load forecasting is rapidly becoming all about handling Big Data, and how it’s going way beyond the spreadsheet systems of yesterday.
I caught up with Ed Cuoco, director, Data Science and Analytics at The Weather Company.
First, what’s the current status of advanced new data analytics tools and data scientists to improve forecasts for electricity consumption?
Cuoco said: “The sophistication and refinement in demand forecasting has increased by leaps and bounds over the last 5 years. The near-real time forward demand curve has become omnipresent within the demand, generation and merchant worlds. Today, the value of forward analytics lies in improving marginal accuracy and timeliness as well as including more diverse and volatile data sources (weather, renewable generation, etc) and applying the insight into more complex business problems where increased speed, accuracy and precision become ever more important.
“Indeed, forecasts have moved beyond capital planning, pricing auctions and energy to become critical across the energy infrastructure, from portfolio management and risk mitigation through demand management and even near-real time operational efficiency.”
So, why is there so much interest in data scientists?
He added that load data and related data sets are seen as core business assets; extracting the value of this data has become a critical task across energy-related businesses. This increased use also increases the criticality of contextual data quality (i.e., is load data that was good enough for thinking about yearly capital spend also good enough for near-real time trading?).
The role of the data scientist lies in the nexus of these two tasks; unlocking more value from diverse data sets (load data, customer data, operational data) while also helping utilities, ISOs, traders and other stake holders understand and overcome limitations in the data themselves to continue to drive more refined understanding and action.
Next, we need to know what sort of analytics tools does The Weather Company work on/with, and what can those do?
Cuoco said: “We use most of the common analytical tools in the market, but what differentiates us is our underlying data platform and its ability to handle huge amounts of weather and load data. Every day The Weather Company maps 62 vertical miles of the atmosphere, all around the globe, to deliver 35+ billion forecasts along with 100+ terabytes of data.
“Four years ago, The Weather Company went through a transformation to a new cloud-based, cloud-agnostic, data-driven infrastructure. That “SUN” platform is an efficient Data as a Service (DaaS) platform that reliably handles these incredible workloads, leveraging 249 different open source tools with proprietary capabilities.
“Most of the platform was written in Scala, and a few of the technologies it leverages include Cassandra, Spark, Riak, and Redis. This platform allows us to turn Big Data into better decision making and provide demand forecasts to energy traders and utilities.
Finally, how is load forecasting now becoming all about handling Big Data?
The robustness, availability and quality of data sets varies tremendously from firm to firm and geography to geography, so there’s a lack of standardization and maturity. In some cases, load data is flowing in near real time at the meter level, in others, only at the grid level, and some in between.
With the implication that all the analytics are the same in principal, the real value to the industry is unlocking the value in the data sets as they exist now, not simply saying “when everything is robust, pristine and perfect, then you’ll see the value.”
What questions can be answered and how confident can one be in that answer is the crux of the big data question in this space. At the same time, load forecasting plays a central role across the energy and utility space, helping drive everything — from real-time demand response to optimizing the use of existing grid infrastructure and preventing catastrophic failures.
These predictions look minutes ahead for traders and days ahead for an ISO or utility, and the business needs to switch seamlessly between geographic and temporal filters. As the questions become more complex, interval meter data is critical but insufficient. Complex weather data, sensor data and customer behavior data all play a role in truly forecasting load and then applying that forecast to finance, operations and customer service.
And, how is it going beyond the spreadsheet systems of yesterday?
Cuoco concluded: “As analytics begin driving ongoing operations as well as strategic planning, the need to handle volume, speed and complexity dwarfs what even the most complex set of spreadsheets can accomplish while also pushing the limits on the underlying products used to create them.
“Further, when this increased business relevance is combined with the more robust, complex, and diverse data, the constant need to revisit and update even the best models becomes critical; analysis is no longer sufficient and, even now, we’re seeing the combination of advanced statistics and machine learning evolve.
“Soon, companies will need solutions that truly learn, interact and reason. Spreadsheets can’t keep up with the load or the complexity and, when pushed to their limits, increase the chances of critical failures in key analytical processes.”
Friends, now, I’ve been nominated for an award in India! 😉
I have been nominated for the prestigious Indywood Media Excellence Awards, being organized at Ramoji Film City, Hyderabad, on Sept 24th 2016.This event is being organized by Indywood and the Government of Telangana State.
The organizers expect the participation of the chairman/CEO/president/directors of more than 100 most reputed establishments to attend this mega event, along with several Forbes listed billionaires and film personalities.
Wow!! All I can say is a deeply, heart-felt thanks! 🙂
I didn’t even know that a simple guy like me would even get nominated for such an award!! Besides, who would read blogs on semiconductors and electronics in India? About a 1,000 people at best? Or maybe, 2,000-3,000? Surely, I myself do not expect that many people to read my stuff in the first place! 🙂
What I’d like to say is that most of my better work had been overseas, especially with Global Sources. I covered computer products, electronic compoments, electronics and telecom for the various verticals. Along the way, I also developed some knowledge about the various products. I want to personally thank Spenser Au, my first publisher and now CEO, Alfred Cheng, Claudius Chan, John Ng, Daniel Tam, Rajendar Gopinath, along with many others.
Pradeep’s Point was started in 2007 on my return from Hong Kong. As many of you would know, this site changed my life. Also, I picked up seven awards, along with seven other awards for the other technology related sites — four for electronic components, and one each for electronics, semiconductors and telecom.
Now, EEWeb.com wants to feature me!
Today, I received an email from Rob Gunayan, from EEWeb.com, who also congratulated me on the nomination and requested to feature me on EEWeb.
For those unaware, EEWeb.com is a premium electrical/electronics online community, owned by Aspen Labs. The head office is located in Boise Idaho, USA. EEWeb said, “We feature Electrical/Electronics Engineers whom we find interesting, and we thought could be an inspiration to others.”
A double WOW! 🙂
Many thanks to the Indywood Media Excellence Awards committee, and also to the EEWeb.com team. You have certainly made it my day! 🙂
GENBAND is a global leader in real time communications software solutions for service providers, enterprises, independent software vendors, systems integrators and developers in over 80 countries. It recently released the Kandy cloud communications solution.
How does Kandy enhance software applications with cloud-based features?
Carlos Aragon, director Kandy/UC Solutions Marketing at GENBAND, said: “The key term is not ‘cloud-based features’, but ‘cloud-based real-time communications’. Kandy is a communications platform-as-a-service (cPaaS) that provides APIs and SDKs to allow communication service providers (CSPs), enterprises and independent software vendors (ISVs) to embed communications within their applications and business processes. Kandy allows these players to make communications an integral part of their applications.
“For example, if you are working on your CRM platform and you need to contact a customer, typically, you would have to find the phone number and call the customer from your phone or softphone. Wouldn’t it be better to just click on the customer name in the screen and have that call happen automatically from within the CRM application? That’s exactly what Kandy enables, your applications can now trigger voice or video calls, messaging, co-browsing, collaboration and many other communication related activities, and you don’t need to be a communications engineer to make that happen or have an expensive communications network to terminate the calls.
Hasn’t it been long claimed that real-time communications will revolutionize the telecoms and technology worlds? Is this any closer to that?
He agreed that the the term revolution is very frequently abused. “We prefer to use evolution. For us, it is the natural step in ICT, first the integration of information technology and communications happened on the network level, by migrating voice and video to IP and using the same transport network to deliver them. We moved from dedicated telephone wiring in our offices to sharing the same Ethernet cables to connect our phones and computers and the analogue and PRI trunks have given way to SIP trunks over an IP link.
“The next step on this evolution was to have the integration at the application level. Real-time communications have already been consumed as applications for years (Skype, WhatsApp, Yahoo Messenger, MSN. Google Talk, and the plethora of Unified Communications Soft Clients in the market), but true integration of those components into business flows and applications never happened.
“Kandy bring us closer to that world, where communication is a natural part of our everyday applications. where a phone line is not tied to a physical telephone, where one can start a conversation with anyone, no matter which device (old and new) they are using from the device or application we choose. That’s what Kandy enables.”
SaaS and PaaS are also related. How is Kandy really different?
He noted that SaaS, PaaS and Kandy are related indeed. “Kandy is PaaS, more specifically, cPaaS (communications platform-as-a-service). With SaaS you obtain a fully functional application that you consume from the cloud (Salesforce.com, Workday, SAP C4C, even Facebook can be qualified as SaaS). With PaaS you use a cloud service as building material to build your final application.
“You can’t just start using PaaS because it doesn’t provide you with an application to consumer directly, instead, it provides you with an API and/or SDK to allow you to use its services within your application. But you have to develop on top of it. In the end, the difference between SaaS, PaaS, IaaS and on premise solutions is who is responsible for each part of the solution.”
IHS Markit Research recently released the ‘Top Consumer and Mobile MEMS Sensor Suppliers for 2015.’
According to IHS, the consumer and mobile micro-electrical mechanical systems (MEMS) sensor revenue declined by 3.4 percent in 2015 with slow growth expected to start from 2017. While microphones and other MEMS categories grew, other categories declined — most notably, motion sensors.
The three leading consumer and mobile MEMS sensor suppliers globally, based on 2015 revenue, were STMicroelectronics at 17.85 percent, Knowles at 17.32 percent, and InvenSense at 17.26 percent.
Sensor companies have seen the reality of a slowing MEMS market and have been working to expand their offerings and business models
What are the reasons for the decline — most notably, motion sensors? As per IHS Markit analyst, Marwan Boustany: “Simply, it is the fact that the ASP decline is greater in magnitude than unit growth. Unit growth is linked directly with slowing end product growth rates (smartphones and tablets specifically) and sensor penetration trends.”
How are sensor companies working to expand their offerings and business models in a slowing MEMS market?
He added: “InvenSense is a good example here. It is entering the ultrasonic fingerprint segment in 2017. It also acquired the ADI MEMS microphone business at end of 2013. InvenSense started selling software/processing as a service with its Coursa range of products (retail and sport) in 2015. It is also giving SensorStudio for free to customers to enable easier sensor based product design in the fragmented and growing IOT space.”
The top three folks share 52+ percent. Who are the other key players and what is their status?
Two other sensor suppliers after the top three are Bosch Sensortec and Goertek.
Bosch Sensortec is a successful sensor supplier with high volumes in inertial sensors (accelerometers, 6-axis IMU, pressure sensor) and growing volume in MEMS microphones through its Akustica business. It has a wide range of customers for its motion sensors from Apple and Samsung to Huawei, and faces stiff competition from InvenSense and STMicroelectronics.
Goertek Inc. is a MEMS microphone supplier with very healthy unit and revenue growth, thanks in large part to its Apple design wins. As of 2015, Goertek still needs to find other large design wins to reduce over reliance on Apple.
I am ever grateful to Dr. Robert N. Castellano, president of The Information Network., USA, for sharing his article titled “Korea’s Capex Holds The Key To Strong Semiconductor Equipment Sales, But Brexit May Weigh On Business,” which is featured on Seeking Alpha.
- A joint study conducted by Applied Materials and Gartner points to semiconductor equipment sales in Korea of $10 billion per year.
- Although US semiconductor equipment companies stand to benefit, numerous Korean equipment companies compete in several sectors with US companies.
- Revenue and growth could be negatively impacted by the strong US dollar and a possible global slowdown from Brexit.
Wafer fabrication equipment (WFE), he says, presents a lucrative market for Korea. In the short term, the strong dollar, as a result of Brexit, will impact sales of WFE equipment to the European semiconductor manufacturers. Traditionally, Europe represents about 5 percent of WFE sales. Hence, Korea could represent an even a greater percentage of WFE sales in 2016.
On the other hand, the turmoil in global financial markets, as a result of Brexit, could have an overall dampening effect on the Korean economy so that revenues generated from Korean semiconductor companies could be impacted. This will result in lower WFE sales as the NAND and smartphone markets slow, particularly in Europe.
He concludes that the US Dollar and Japanese Yen, could strengthen throughout 2016 and impact revenue growth for US and Japanese equipment companies selling in Korea.
It would be interesting to see how H2-2016 pans out!
Let’s turn our attention to semiconductors. The key question: how is the industry performing?
Now, the Semiconductor Industry Association (SIA) has put it as $26 billion for May 2016, an increase of 0.4 percent compared to previous month’s total of $25.9 billion, but a decrease of 7.7 percent compared to the May 2015 total of $28.1 billion.
If we look at SEMI’s North American semiconductor equipment industry figures,
the three-month average of worldwide bookings in June 2016 was $1.71 billion. The bookings figure is 2.1 percent lower than final May 2016 level of $1.75 billion, and 12.9 percent higher than June 2015 order level of $1.52 billion.
However, IC Insights has lowered its 2016 semiconductor market forecast to -1 percent. It says, ” Rarely can there be strong semiconductor market growth without at least “good” worldwide economic growth to support it. ”
Besides the impact of Brexit on the UK, and how it can impact the European semiconductor industry, the DRAM industry is said to register a 19 percent drop of $8.5 billion this year to $36.5 billion. The semiconductor market growth, excluding DRAM, is forecast to be +2 percent for 2016.
As the SIA puts it, “additional research is needed as we transition to an even more connected world, enabled by innovations like the IoT.”
According to the SIA, research is needed in areas such as energy-efficient sensing and computing, cyber-physical systems, intelligent storage, real-time communication ecosystem, multi-level and scalable security, next-generation manufacturing paradigms, insight computing, and an IoT test platform.
Why can’t today’s computers process data like the brain? Brain-inspired computing has been a focus area. How that will pan out would make interesting reading.
Here’s my last piece for Control Engineering Asia on what should we expect in 2016?
There are certain factors such as the adoption of IoT, advancement in M2M, the increasing demand of robots in the industrial sector, etc., that can result positively on the industrial control and factory automation market.
The global industrial automation services market is likely to exceed $50 billion by 2019. The industrial controls system market is likely to reach $81 billion in 2021!
For industrial controls/valves, the Asia-Pacific market for industrial valves is projected to grow at 6.88 percent CAGR between 2014 and 2019. The industrial control system (ICS) security market is likely to grow at a CAGR of 7.6 percent from 2014 to 2019.
How about PLC/DCS and SCADA? One observer said that the PLC market in Europe is likely to grow at a CAGR of 7.44 percent from 2014-2019. The Asia-Pacific market for PLCs is likely to hit $3.6 billion in 2016. Also, the APAC DCS (decentralized automated control systems) market is set to grow at 4.36 percent CAGR over 2014-2019.
The Southeast Asia supervisory control and data acquisition (SCADA) market is likely to double in five years from $148.7 million in 2012 to reach an estimated $274.8 million in 2018. For PLM, the market is likely to reach $75.87 billion by 2022. The CAD market in aerospace and defense sector in North America is set to grow at a CAGR of 4.2 percent over 2014-2019.
IoT to dominate?
Nearly everyone’s favorite subject happens to be the IoT. The industrial IoT market is likely to grow at 26.56 percent CAGR to 2019. The M2M connections market will likely be worth $35.16 billion by 2020. Even the IoT in retail market is said to be worth $35.64 billion by 2020.
The installed base of wireless M2M devices in the oil & gas industry alone is set to grow at CAGR of 20.1 percent from 0.5 million units at end of 2014 to 1.25 million units by 2019.
How about sensors? Asia Pacific is set to reach $5.23 billion by 2020 at CAGR of 4.09 percent. In water/wastewater, the water recycle market and water re-use market growth are expected to be at 22 percent and 19 percent CAGR to 2019.
Hope some or all of these targets are met in 2016.