A study led by scientists from the University of Chicago, in collaboration with New York University and the University of California Santa Cruz, has found that neural networks cannot forecast weather events beyond the scope of existing training data, which might leave out weather that’s unprecedented in recorded history such as 200-year floods, unprecedented heat waves or massive hurricanes.
According to their research, published in Proceedings of the National Academy of Sciences, this is because neural networks only predict based on patterns from the past. The researchers highlight that this limitation is particularly important as researchers incorporate neural networks into operational weather forecasting, early warning systems and long-term risk assesments. However, they also said there are ways to address the problem by integrating more math and physics into the AI tools.
“AI weather models are one of the biggest achievements in AI in science. What we found is that they are remarkable, but not magical,” said Pedram Hassanzadeh, an associate professor of geophysical sciences at UChicago and a corresponding author on the study. “We’ve only had these models for a few years, so there’s a lot of room for innovation.”
Gray swan events
For weather forecasts, scientists train neural networks by feeding them decades’ worth of weather data. Then a user can input data about the current weather conditions and ask the model to predict the weather for the next several days.
The AI models can achieve the same accuracy as a supercomputer-based weather model that uses 10,000 to 100,000 times more time and energy, Hassanzadeh said.
“These models do really, really well for day-to-day weather,” he commented. “But what if next week there’s a freak weather event?”
The team’s concern is that the neural network is only working off the weather data we currently have, which goes back about 40 years. However, that’s not the full range of possible weather.
“The floods caused by Hurricane Harvey in 2017 were considered a once-in-a-2,000-year event, for example,” Hassanzadeh said. “They can happen.”
The team decided to test the limits of the AI models using hurricanes as an example. They trained a neural network using decades of weather data but removed all the hurricanes stronger than a Category 2. Then they fed it an atmospheric condition that leads to a Category 5 hurricane in a few days. They found that the model couldn’t extrapolate to predict the strength of the hurricane.
“It always underestimated the event. The model knows something is coming, but it always predicts it’ll only be a Category 2 hurricane,” said Yongqiang Sun, research scientist at UChicago and the other corresponding author on the study.
Hurricane warnings and why physics matters
The big difference between neural networks and traditional weather models is that traditional models “understand” physics. Scientists design them to incorporate the current understanding of the math and physics that govern atmospheric dynamics, jet streams and other phenomena.
The neural networks aren’t doing any of that. Like ChatGPT, which is essentially a predictive text machine, they simply look at weather patterns and suggest what comes next, based on what has happened in the past.
Hassanzadeh highlights that no major service is currently using onlyAI models for forecasting, but as their use expands, this tendency will need to be factored in.
However, they found the model could predict stronger hurricanes if there was any precedent, even elsewhere in the world, in its training data. For example, if the researchers deleted all the evidence of Atlantic hurricanes but left in Pacific hurricanes, the model could extrapolate to predict Atlantic hurricanes.
“This was a surprising and encouraging finding: it means that the models can forecast an event that was unpresented in one region but occurred once in a while in another region,” Hassanzadeh said.
Merging approaches
The solution, the researchers suggest, is to begin incorporating mathematical tools and the principles of atmospheric physics into AI-based models.
“The hope is that if AI models can really learn atmospheric dynamics, they will be able to figure out how to forecast gray swans,” Hassanzadeh said.
One promising approach the team is pursuing is called active learning – where AI helps guide traditional physics-based weather models to create more examples of extreme events, which can then be used to improve the AI’s training.
“Longer simulated or observed datasets aren’t going to work. We need to think about smarter ways to generate data,” said Jonathan Weare, professor at the Courant Institute of Mathematical Sciences at New York University and study co-author.
“In this case, that means answering the question ‘where should I place my training data to achieve better performance on extremes?’ Fortunately, we think AI weather models themselves, when paired with the right mathematical tools, can help answer this question.”
In related news, a team of researchers from the University of Pennsylvania and Microsoft Research recently developed Aurora, a machine-learning model with predictive capabilities for air quality, ocean waves, tropical cyclone tracks and weather. Click here to read the full story
