TLDR: This research paper proposes that successful time-series forecasting models must learn the underlying data dynamics. Through a new PRO-DYN nomenclature, the authors analyze existing models, finding that top-performing architectures incorporate a learnable dynamics block, ideally as the final predictor. Experiments confirm that adding learnable dynamics significantly improves model performance, highlighting its critical role in accurate time-series predictions.
In the rapidly evolving landscape of artificial intelligence, where boundaries between different data types are increasingly blurred, one area continues to pose a significant challenge: time-series forecasting. Despite the success of advanced deep learning models in various fields, simpler models often outperform them when it comes to predicting future trends in time-series data. This intriguing observation has led researchers to hypothesize that the key to accurate time-series forecasting lies in a model’s ability to learn the underlying ‘dynamics’ of the data.
A recent research paper, titled “Dynamics is what you need for time-series forecasting!” by Alexis-Raja Brachet, Pierre-Yves Richard, and Céline Hudelot, delves into this very hypothesis. The authors propose that unlike text generation, which many current models are based on, time-series generation follows a different mechanism—one governed by an evolution law, or a dynamical system. Therefore, time-series forecasting models should be designed to replicate this mechanism by learning the data’s inherent dynamics.
Understanding PRO-DYN: A New Lens for Analysis
To systematically investigate their hypothesis, the researchers developed an original framework called the PRO-DYN nomenclature. This framework categorizes the computations within time-series forecasting models into two types: PRO (PROcessing) functions and DYN (DYNamics) functions. PRO functions handle data transformations within the same time interval, such as normalization or feature extraction. DYN functions, on the other hand, are responsible for predicting future states based on current ones, essentially modeling the time evolution of the data.
Through this new lens, the authors conducted a systemic study of existing time-series forecasting models. Their analysis revealed two critical observations. Firstly, models that performed poorly either lacked learnable dynamics modeling capabilities entirely or only had them partially. Secondly, state-of-the-art architectures consistently incorporated a learnable dynamics block, and its placement at the very end of the model, acting as the final predictor, was of prime importance.
Experimental Validation: Adding Dynamics and Optimal Placement
To validate these observations empirically, the researchers designed two sets of experiments. The first set (RQ1) aimed to determine if adding a full learnable dynamics component could enhance model performance. They took several popular models, including Informer, FiLM, MICN, and FEDformer, which originally had limited or non-learnable dynamics, and incorporated a simple linear dynamics layer into them. The results were compelling: the ‘DYN added’ versions of these models showed tangible performance improvements, especially Informer and FiLM, strongly supporting the idea that learnable dynamics capabilities drive performance.
The second set of experiments (RQ2) investigated the optimal placement of the dynamics block. The authors modified well-performing foundation models like iTransformer, PatchTST, and Crossformer by adding a linear dynamics layer at the beginning, effectively turning their original dynamics layer into a post-processing unit. The outcome was clear: for PatchTST and Crossformer, this ‘post-processing’ configuration led to a performance drop, confirming that a pre-processing-like architecture, where the dynamics block acts as the final predictor, is generally superior. While iTransformer showed some resilience, the overall trend reinforced the importance of dynamics being the final step in the prediction process.
Also Read:
- Assessing Time Series Forecastability with Spectral and Lyapunov Measures
- Predicting the Future: How Frozen Video Models Learn to Forecast
Beyond the Experiments: The Path Forward
An ablation study was also conducted to ensure that the observed performance gains were indeed due to the learnable dynamics and not merely from added parameters or changes in data length. The results largely confirmed that the performance improvements were primarily driven by the models’ enhanced ability to learn dynamics.
This research provides a fresh perspective on time-series forecasting, emphasizing the fundamental role of learnable dynamics. It suggests that future model designs should prioritize incorporating robust dynamics learning capabilities, ideally positioned as the ultimate predictor. While this study focused on linear dynamics, the authors propose exploring other dynamics functions, such as autoregressive mechanisms, and further investigating State-Space Models (SSMs) through the PRO-DYN nomenclature. The work underscores that while dynamics is crucial, other factors like PRO function backbones and temporal dimension computations also play a role in overall performance.
For a deeper dive into the methodology and detailed experimental results, you can access the full research paper here.
