Forecast accuracy is the one claim buyers cannot easily check and accurate providers cannot easily prove. A newly published calibration shows what it takes to settle the question, and in public.
A weather forecast is one of the few products bought at scale whose central claim cannot readily be checked by the buyer. Verification exists at every provider of any size, but it is self-reported, built on different station pools, error statistics and time windows, and it is therefore not comparable between vendors. Two consequences follow. An organization choosing a forecast supplier is left weighing marketing material against the cost of building verification in house, and a provider that is genuinely the most accurate in its segment has no neutral way to demonstrate it. That second problem is the older one. Measuring accuracy and being able to declare it publicly are different tasks, and the industry has spent two decades solving only the first.
The Most Accurate Forecast Awards, announced on August 3, are an attempt at the second: an annual, independently calculated result naming the most accurate provider in each combination of market, region, metric and forecast horizon, drawn from forecasts scored against quality-controlled surface observations since 2003. The full methodology has been published, and the design decisions in it are more instructive than the winners.
Why the scale is the hard part
Mean absolute error will say which of two temperature forecasts came closer to the observation. It says nothing about whether a 3.5 ° temperature error is better or worse than an Equitable Threat Score of 0.367 on precipitation. Any benchmark spanning more than one forecast element has to reconcile those units, and the reconciliation is a choice of scale rather than a measurement.
The conventional route is a skill score against a reference forecast, usually persistence or climatology. Tested against the archive, it placed the strongest providers in the fifties to seventies, understating how accurate leading forecasts have become. Using persistence to define the bottom of the scale failed outright. It is a strong baseline in parts of the data and beats real provider forecasts on some elements, wind in particular, which produced anchor points that were not monotonic in 211 of 1,386 calibration cells.
What replaced it is percentile anchoring. For each combination of metric, horizon and region, the distribution of provider errors across 2021 through 2024 fixes four points: zero error scores 100, the fifth percentile of observed provider losses scores 92, the median scores 72, and the 90th percentile scores 40. Because every cell is anchored to its own history, a score of 85 means the same thing whether the metric is temperature in North America or precipitation in Asia-Pacific.
Freezing the curve
The calibration is derived once and applied unchanged to every award period. Refitting it annually would be the intuitive choice and would quietly destroy the scale, because industry-wide improvement would be normalized away each year: a provider could improve its forecasts measurably and watch its score fall because the field improved faster. With the curve fixed, a rising score means the forecasts genuinely got better, and the best provider in a given year is not automatically awarded 100. Recalibration, when the industry outgrows the baseline, ships as an announced new version, and every archived result records the version under which it was scored.
Keeping the comparison honest
An earlier generation of the system combined metrics by rank averaging. Modeling against historical data showed that this rewards consistency rather than accuracy: in roughly 44% of composite categories, the rank-averaged winner had a lower accuracy score than the provider it beat. Averaging the calibrated scores instead means the published number and the published ranking cannot contradict each other. The eligibility rules close the remaining gaps. A provider needs at least 1,000 scored forecasts in the relevant cell, and a qualifying score for every day in the award span, so a provider forecasting seven days ahead is absent from a 14-day award rather than averaged across the portion it covers. Overnight low awards stop at day 13, because a day-14 low verifies the following morning.
What a public benchmark changes
For forecast producers, the effect is that accuracy becomes something that can be stated rather than asserted. Several of the inaugural winners are not consumer-facing names, and would not have surfaced in any ranking driven by brand recognition. For the organizations that depend on forecasts, in energy trading, aviation, insurance and agriculture, an annual independent result narrows a procurement question that vendor material has never been able to settle.
The larger value accumulates rather than arrives. Machine learning models are moving from research into operations faster than the evidence for their operational skill is being published, and a yardstick applied on the same basis every year is one of the few practical ways to separate real gains from confident marketing. The inaugural awards cover June 2025 through May 2026, and the full ranked field for every category is published alongside the winners.
About the author
Eric Floehr is the founder and chief executive of ForecastWatch, which he established in 2003 to measure the accuracy of operational weather forecasts. Working from Columbus, Ohio, he has spent more than two decades building and maintaining the archive of forecasts and matched observations that underpins the company’s verification work, and he leads the methodology behind its accuracy scoring.