Sourcing Global Weather Data for Precision Forecasting in the Private Sector
by Mike Eilts, on Oct 4, 2018 1:22:16 PM
In my previous blog, I wrote about the vital partnership between the National Weather Service and private weather companies and how each brings an essential function to the weather enterprise. In this blog, I’d like to tell you more about how a private weather company, such as WDT, builds forecasts from a variety of sources from around the world.
Gathering Raw Data
WDT collects and aggregates data from the National Weather Service and many other sources. The NWS makes its data available for free, though there may be a small cost to obtain and transfer the data. The information we collect from the NWS includes radar data, satellite data, surface observations, and model information. A few of the more notable model forecasts that we acquire from the NWS are the GFS (Global Forecast System) and the NAM (North American Mesoscale). The NWS also provides free forecasts, but we choose to produce our own because ours are more accurate and are also for locations beyond just the United States.
We also produce forecasts for unique parameters that are important to our clients. These include forecasts for winds at 80 and 100 meters above the ground (for wind farms and tall structures), evapotranspiration (for our agriculture clients), and site-specific forecasts for other parameters that create value for our customers.
We gather additional satellite, observation and model data—at varying costs—from several other sources located mainly outside the United States. We spend significant dollars to acquire one of the world’s most respected weather models, the ECMWF (European Center for Medium-Range Weather Forecasts). We also pay to obtain weather information from countries such as Canada, Australia, New Zealand, and Japan, to name a few. We buy lightning data from a private company while obtaining additional weather data from the World Meteorological Organization. In total, a private weather company that provides global forecasts and services can spend millions of dollars per year to acquire the raw ingredients needed to make forecasts for global customers.
Data Modification and Bias Correction
We aggregate all of these data in our cloud computing infrastructure, where we clean it and remove various flaws and biases that would otherwise skew the data and produce inaccuracies from the start. Although weather models are continually improving, they still have limitations, many of which we know about because of our extensive experience. For example, we know that certain models over the Gulf of Mexico may forecast wind speeds too low after a cold frontal passage. After removing biases from each of the individual models, we combine forecasts from multiple models using machine learning techniques that constantly get better over time.
We also run our own forecast models at many locations all over the world, often at very high resolution (1-3 miles), so that we can provide rapidly updated and accurate forecasts to our customers and to our meteorologists who are providing services to all of our customers.
We perform this massive data aggregation and expert curation of the forecasts twice each day. It allows us to produce accurate forecasts around the globe for approximately 10,000 client locations at the push of a button.
Modifications for Special Forecasts
Beyond traditional forecasts, we gather radar data to produce individual short-term precipitation predictions, such as where rain is going to happen (as well as how much has fallen in the last hour or 24 hours), where hail may be occurring (and the path it may follow), and where tornadoes may strike.
As with model data, radar data often requires modification and machine learning, where our system learns which data are valid and meaningful, and which are not. For example, radar often produces what we call clutter, false returns that can appear as precipitation but result from mountains, buildings, atmospheric disturbances, and birds and insects in proximity to the radar. Our expert radar team has spent substantial time over the years building automated algorithms that clean up the data so that our final product is a radar mosaic that’s reliable and provides accurate information.
A Massive Amount of Data
Beyond the financial cost of gathering the “back end” data required for a forecast, a private weather company must have the capability to house large amounts of data. At WDT, we handle approximately one petabyte per month. (A petabyte is a million gigabytes or a thousand terabytes.) For all our skill with the science of meteorology, we are also truly a big data company.
Access to Vast Expertise for a Small Cost
As you begin to understand all the work that goes into producing forecasts—including the costs, computing power, and human resources—you get a sense as to why so few companies bring weather guidance in-house. It takes a significant investment, massive infrastructure, and experienced expertise to provide value. Companies that choose to contract with a private forecasting company buy access to meteorologists available around the clock, as well as features such as portals and apps that provide access to the customized products they need—for a fraction of what it would cost if they tried to do it themselves.
Gathering Raw Data
WDT collects and aggregates data from the National Weather Service and many other sources. The NWS makes its data available for free, though there may be a small cost to obtain and transfer the data. The information we collect from the NWS includes radar data, satellite data, surface observations, and model information. A few of the more notable model forecasts that we acquire from the NWS are the GFS (Global Forecast System) and the NAM (North American Mesoscale). The NWS also provides free forecasts, but we choose to produce our own because ours are more accurate and are also for locations beyond just the United States.
We also produce forecasts for unique parameters that are important to our clients. These include forecasts for winds at 80 and 100 meters above the ground (for wind farms and tall structures), evapotranspiration (for our agriculture clients), and site-specific forecasts for other parameters that create value for our customers.
We gather additional satellite, observation and model data—at varying costs—from several other sources located mainly outside the United States. We spend significant dollars to acquire one of the world’s most respected weather models, the ECMWF (European Center for Medium-Range Weather Forecasts). We also pay to obtain weather information from countries such as Canada, Australia, New Zealand, and Japan, to name a few. We buy lightning data from a private company while obtaining additional weather data from the World Meteorological Organization. In total, a private weather company that provides global forecasts and services can spend millions of dollars per year to acquire the raw ingredients needed to make forecasts for global customers.
Data Modification and Bias Correction
We aggregate all of these data in our cloud computing infrastructure, where we clean it and remove various flaws and biases that would otherwise skew the data and produce inaccuracies from the start. Although weather models are continually improving, they still have limitations, many of which we know about because of our extensive experience. For example, we know that certain models over the Gulf of Mexico may forecast wind speeds too low after a cold frontal passage. After removing biases from each of the individual models, we combine forecasts from multiple models using machine learning techniques that constantly get better over time.
We also run our own forecast models at many locations all over the world, often at very high resolution (1-3 miles), so that we can provide rapidly updated and accurate forecasts to our customers and to our meteorologists who are providing services to all of our customers.
We perform this massive data aggregation and expert curation of the forecasts twice each day. It allows us to produce accurate forecasts around the globe for approximately 10,000 client locations at the push of a button.
Modifications for Special Forecasts
Beyond traditional forecasts, we gather radar data to produce individual short-term precipitation predictions, such as where rain is going to happen (as well as how much has fallen in the last hour or 24 hours), where hail may be occurring (and the path it may follow), and where tornadoes may strike.
As with model data, radar data often requires modification and machine learning, where our system learns which data are valid and meaningful, and which are not. For example, radar often produces what we call clutter, false returns that can appear as precipitation but result from mountains, buildings, atmospheric disturbances, and birds and insects in proximity to the radar. Our expert radar team has spent substantial time over the years building automated algorithms that clean up the data so that our final product is a radar mosaic that’s reliable and provides accurate information.
A Massive Amount of Data
Beyond the financial cost of gathering the “back end” data required for a forecast, a private weather company must have the capability to house large amounts of data. At WDT, we handle approximately one petabyte per month. (A petabyte is a million gigabytes or a thousand terabytes.) For all our skill with the science of meteorology, we are also truly a big data company.
Access to Vast Expertise for a Small Cost
As you begin to understand all the work that goes into producing forecasts—including the costs, computing power, and human resources—you get a sense as to why so few companies bring weather guidance in-house. It takes a significant investment, massive infrastructure, and experienced expertise to provide value. Companies that choose to contract with a private forecasting company buy access to meteorologists available around the clock, as well as features such as portals and apps that provide access to the customized products they need—for a fraction of what it would cost if they tried to do it themselves.
