TLDR: This research paper evaluates various machine learning and deep learning models, including transformers and large language models, for forecasting PM2.5 air pollution levels in Bucharest, Romania. Using a four-year hourly dataset, the study found that transformer models generally outperformed other architectures, especially for longer prediction horizons, with advanced recurrent neural networks and ensemble methods also showing strong performance. The work highlights the potential of these models for providing early warnings and aiding environmental decision-making in urban areas.
Air pollution, particularly fine particulate matter (PM2.5), poses a significant health risk in urban environments worldwide, including Bucharest. Exposure to PM2.5 is linked to various health issues, from respiratory and cardiovascular diseases to more severe conditions like cancer and premature death. The ability to accurately forecast these pollution levels can provide crucial early warnings, enabling governments and environmental organizations to make informed decisions and raise public awareness.
A recent research paper, AIRPOLLUTIONFORECASTING INBUCHAREST, delves into the development and evaluation of various machine learning and deep learning models to predict future PM2.5 levels in Bucharest. The study, conducted by Dragos, -Andrei S, erban, Răzvan-Alexandru Smădu, and Dumitru-Clementin Cercel from the National University of Science and Technology POLITEHNICA Bucharest, aimed to assess and compare the performance of a wide range of models, from traditional linear regression to advanced recurrent neural networks, transformers, and even large language models, for forecasting PM2.5 over different time horizons (1, 2, and 4 hours ahead, and up to 8 hours for some deep learning models).
The Bucharest Dataset: A Closer Look
The researchers utilized a comprehensive dataset collected hourly from August 1, 2019, to July 31, 2023, at the B-1 RNMCA BUC station near Mill Lake in Bucharest. This dataset includes not only PM2.5 and PM10 concentrations but also other pollutants like NO2, SO2, CO, and O3, alongside meteorological data such as air temperature, wind speed and direction, and humidity. This extensive four-year record provided a rich foundation for training and testing the models.
Before feeding the data into the models, a meticulous data cleaning and processing phase was undertaken. This involved identifying and handling outliers using the forward-backward exponential weighted moving average (FBEWMA) algorithm, interpolating missing values, and converting any faulty negative measurements to zero. To enhance the models’ predictive power, new features were engineered, including lag features for PM2.5 and other pollutants, as well as categorical variables for the day of the week, weekend status, and season. These additions helped the models better understand temporal patterns and external influences on air quality.
A Spectrum of Models Explored
The study cast a wide net, evaluating an impressive array of forecasting models:
- Traditional Machine Learning: This category included linear regression variants (Lasso, Ridge, ElasticNet), Support Vector Regression (SVR), and ensemble methods like Random Forest and XGBoost.
- Statistical Time Series Models: ARIMAX and SARIMAX, which are extensions of ARIMA models, were used to capture autoregressive, integrated, and moving average components, with SARIMAX specifically addressing seasonal patterns.
- Deep Learning Architectures: The researchers explored Multi-Layer Perceptrons (MLP), Kolmogorov-Arnold Networks (KANs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks (including stacked and bidirectional versions, and encoder-decoders), Gated Recurrent Units (GRU) networks, and Convolutional Neural Networks (CNNs), along with various hybrid combinations.
- Transformers for Time Series: Advanced transformer models specifically designed for time series forecasting, such as Darts Transformer, Informer, and PatchTST, were a key focus.
- Large Language Models (LLMs): Even T5-small and T5-base language models were fine-tuned for the forecasting task, both with and without Retrieval-Augmented Generation (RAG) to incorporate past data.
Key Findings: Transformers Lead the Way
The quantitative analysis revealed that transformer models consistently delivered the best overall performance across various forecasting horizons. Informer excelled in longer-term predictions, while PatchTST showed superior accuracy for short-term forecasts. These models demonstrated a remarkable ability to capture both long-range and short-term dependencies in the time series data without relying on recurrence.
Following closely were advanced RNN-based architectures and hybrid models. The LSTM encoder-decoder, for instance, showed strong performance, explaining 86% of the variability for 1-hour forecasts. Ensemble methods like Random Forest and XGBoost also yielded promising results among the classical machine learning models, particularly for short-term predictions, demonstrating that combining multiple learners can significantly enhance accuracy. Interestingly, a hybrid KAN-MLP model, which first captures local patterns before feeding them into MLP layers, outperformed its MLP-KAN counterpart.
While large language models like T5 showed decent performance for short-term horizons, they were generally outperformed by other deep learning models for longer prediction periods, and the integration of RAG provided only limited improvements. Linear regression models, as expected, were the least effective, highlighting the complex, non-linear nature of air pollution dynamics.
Also Read:
- Predicting Transit Delays: How AI Learns from Unstructured Alerts
- Real-World Data Assimilation: Introducing the DAMBench Framework
Looking Ahead
This research underscores the potential of advanced machine learning and deep learning, especially transformers, in providing accurate air pollution forecasts for urban areas like Bucharest. The study also acknowledges limitations, such as using data from a single station and the absence of other influential factors like real-time traffic or industrial activity. Future work aims to address these by incorporating data from multiple stations and additional environmental factors, potentially leading to even more robust and comprehensive air quality prediction systems.
