As our changing climate fuels more frequent wildfires, there’s a growing concern about the health impacts of particulate matter (PM2.5) pollution, especially among people with pre-existing respiratory conditions. (Source: South Coast Air Quality Management District)
3 Questions: Evaluating hourly air pollution forecasts in a time of more frequent wildfires
MIT researchers show where today’s models fall short
As our changing climate fuels more frequent wildfires, there’s a growing concern about the health impacts of particulate matter (PM2.5) pollution, especially among people with pre-existing respiratory conditions. Hourly air quality forecasts could help these individuals determine whether and when to go outdoors. But are existing forecasts good enough to be used for these decisions? A recently published paper introduces a new method for evaluating the accuracy and reliability of today’s air quality forecast tools. Here the study’s co-authors, most of whom are affiliated with the MIT Center for Sustainability Science and Strategy—Renato Berlinghieri, David R. Burt and Tamara Broderick from the MIT Laboratory for Information and Decision Systems; and Paolo Giani and Arlene M. Fiore from the MIT Department of Earth, Atmospheric and Planetary Sciences—share some of their findings.
Q: Why do we need accurate and reliable hourly air quality forecasts?
A: Recent research estimates that smoke PM2.5 has already caused over 40,000 annual excess deaths in the U.S., averaged over the period 2011–2020. Smoke PM2.5 can also exacerbate respiratory conditions and lead to emergency room and doctor visits.
Many people rely on weather forecasts to plan when to go outside or whether to carry an umbrella. Analogously, a reliable air quality forecast could allow an individual with a respiratory condition to decide when to complete household chores or whether to work from home — or allow an athlete to plan a run or hike.
Q: How do today’s air quality forecast tools fall short?
A: We looked at six air quality forecasts that have some capability to predict wildfire smoke in the continental United States, including one from a modern AI foundation model. We evaluated forecasts on their ability to answer two questions an individual might have: (1) Should I take precautions at all today? (2) When is the best time to go outside today (e.g. to complete a chore)? In both cases, we compared to a simple baseline of just checking the Environmental Protection Agency's reported local PM2.5 value in the morning and assuming it will be the same all day. A useful forecast should perform better than this baseline by predicting times of higher and lower air pollution.
For the first question, we found that none of the forecasts performed meaningfully better than the simple baseline. Looking at some specific wildfire smoke events, we could see cases where the PM2.5 was already visibly elevated in the morning, but the forecasts just hadn’t caught up and didn’t predict high values of air pollution at any point in the day.
For the second question, we found that a subset of the forecasts performed better than the baseline. That is, some of the forecasts allowed individuals to reduce PM2.5 exposure by going outside at the predicted best time during the day. Interestingly, the AI forecast did not perform as well as the others (and didn’t beat the baseline in this task). That said, forecasts were often inaccurate in predicting peaks and troughs of air pollution during the day, and they haven’t yet incorporated some of the available observations (see below for more detail), so we think there is still room for further improvement.
Q: What steps can be taken to improve the accuracy and reliability of air quality forecasts?
A: We think there is a lot of potential for improvement, but some care is required. Physics-driven forecasts (the five non-AI forecasts) require substantial work to incorporate new data sources. But all five currently exclude important data sources. For instance, the National Oceanic and Atmospheric Administration is actively working to incorporate near-real-time information from surface monitors in its forecasts, and forecasts are often not incorporating information from satellite products that can track the transport of smoke plumes. On the other hand, AI can naturally incorporate new data sources as features. But the AI model we evaluated isn’t trained on wildfire information or ground-truth surface observations. Also, when training AI or choosing a forecast, metrics matter. Fire smoke is not common day-to-day, so forecasts might still perform well on traditional metrics even if they underestimate smoke pollution. Instead, in our work, we propose metrics that are directly tailored to the two individual decision-making tasks above.