Eric Floehr has been working with companies like AccuWeather and The Weather Channel to review the accuracy of their forecasts. His company – ForecastWatch – also works closely with many other organizations to help them model their weather planning to ensure they are best prepared for weather events.
Floehr says the key thing is combining the latest data analytics tools with nearly a billion archived weather forecasts dating back to 2004. The end result is an extremely reliable way of reviewing forecasts from many weather outfits, and understanding how they match up with previous forecasts. To discover more, Digital Journal caught up with Floehr.
Digital Journal: How accurate are weather forecasts, in general?
Eric Floehr: In general, forecasts are very accurate. One day out high temperature forecasts in the United States are within three degrees over 80% of the time. Five day out high temperature forecasts are within three degrees over 60% of the time.
There is measurable precipitation in the 800 locations we measure, approximately 36.5% of the days in 2018. This means if you never forecast precipitation, you would be right 63.5% of the time. Even seven days out, the best forecasters are right as to whether or not there will be precipitation over 70% of the time.
We calculate dozens of other measures of accuracy on many different forecast parameters, in addition to high temperature and precipitation. These are just a few, and these statistics are available on our site ForecastAdvisor.com, which helps people understand the accuracy of weather forecasts for their location in the United States.
DJ: What are the factors that lead to variation with forecasts?
Floehr: For the forecasts themselves, the factors that cause forecasts from different providers to be different include which models the forecasters use, what post-processing (including learning systems and AI) is performed, and what human forecasters change.
Variation in the accuracy of forecasts is caused by a number of factors. Temperature forecasts tend to be more accurate in the summer than in winter, some regions or areas (like the coasts and tropical areas) tend to be easier to forecast temperatures, but the tropics can be difficult to predict precipitation, especially in areas like Florida where small, convective thunderstorms are frequent and hard to predict exactly where they will form.
DJ: How did you develop your latest technological solution?
Floehr: Building the system has been an iterative process over the last 15 years. We try to use agile methods, and both use and contribute back to many open source projects. Our technology stack includes Linux, Python, Django, PostgreSQL, and more.
DJ: Where did you draw the data from?
Floehr: We are fortunate in the United States to have easy access to large amounts of observational data. For other countries, we get our data from a variety of sources. Quality levels vary by country and ensuring the integrity and quality of the data is very important to us.
For forecasts, when we first started out, it was all web scraping, even for forecasting companies that were customers. Eventually some forecasters provided FTP feeds. Now, many of the forecasters provide comprehensive APIs from which we collect forecasts. When an API is not available, we still rely on web scraping.
DJ: How did you test out the system?
Floehr: We use unit tests, and have been increasing our use of test frameworks like pytest. For integration, we hand check the data through the system, and we perform manual calculations to ensure results are correct. We continuously audit the data and correct problems with the information when they are found. Transparency is key, as well as communication, when problems are found.
DJ: Which types of businesses do you work with?
Floehr: We work with the majority of weather forecasting companies, including companies like AccuWeather, The Weather Company (IBM), and Global Weather Corporation, as well as international forecasting companies like Pelmorex. They value our comparative forecast accuracy system that not only allows them to understand the accuracy of their forecasts from an operational and strategic level, but also to communicate that accuracy to customers and partners.
Outside of the weather industry, we help energy companies, transportation companies, media companies, investment firms, and outdoor entertainment/sports firms choose the best forecasts for them as well as to use those forecasts more effectively to make better decisions and better model behavior.
DJ: To what uses is your weather data put by these firms?
Floehr: Forecasting companies use the accuracy data to make strategic business decisions on where to invest in new technology to improve their forecasts, partner with other forecasting companies to augment their capabilities, or make acquisitions to continue to provide the best forecasts they can for their customers. They use our products to help meteorologists on a tactical level better understand where forecasts are good and not-so-good, and helps them as a feedback loop and unbiased monitor.
Externally, our data helps their customers understand the accuracy of their forecasts, and provides a competitive advantage to those forecasting companies who invest in improving the accuracy of their forecasts. In the age of information, we all want the best information possible, and our data shows in an unbiased, scientific manner the quality of the forecast.
Businesses make use of our data in a number of ways. First, we help them select the best weather forecast for their needs. Not only do we help them evaluate the accuracy for the parameters, locations, and time frames they are interested in, we help them understand whether those differences are meaningful for their applications. For some businesses, a half-a-degree difference in average high temperature error is meaningful, and thus valuable, but in other cases, it’s not. We also help businesses continuously monitor the quality and accuracy of the forecasts they do receive, so that they can have peace of mind that their forecasts continue to have the quality they need, and alerts them if there are issues.
Finally, we help businesses better use weather forecasts. We help them understand how to evaluate the uncertainty of a forecast, and work with clients when appropriate to evaluate probabilistic forecasts and do risk analysis. We also know that a large portion of the economy is driven by weather. From providing historical weather forecasts, to working with customers to do analysis and modelling, we help businesses better understand and predict the impacts the weather forecast (and the weather) will have on sales, attendance, usage, and more.
DJ: How would you like to develop the technology further?
Floehr: We would like to expand our system to monitor the quality and accuracy of hourly and nowcast forecasts as well as seasonal forecasts. We would like to greatly expand the monitoring of weather forecasts from the approximately 1200 worldwide locations we have today. We would like to continue to increase the value we provide businesses from the high quality, normalized historical weather forecasts, to worldwide historical observational data to data analytics and to provide tailored solutions to the energy, transportation, and all industries who are impacted by the weather.
