TLDR: TempoPFN is a new time series foundation model that uses linear Recurrent Neural Networks (RNNs) and is trained exclusively on diverse synthetic data. It achieves top-tier zero-shot forecasting performance on the Gift-Eval benchmark, outperforming other synthetic-only models and many real-data trained models. The model’s architecture allows for fully parallelizable training and inference, making it highly efficient, and it demonstrates superior robustness to missing values. The project is fully open-source, providing a reproducible foundation for future research in time series forecasting.
A new research paper introduces TempoPFN, a groundbreaking univariate time series foundation model designed for zero-shot forecasting. This model leverages linear Recurrent Neural Networks (RNNs) and is pre-trained exclusively on synthetic data, addressing critical challenges in efficient long-horizon prediction and reproducibility in the field of time series forecasting.
Traditional approaches to time series forecasting often struggle with long-horizon predictions due to computational complexity, especially with Transformer-based models. While non-linear RNNs can maintain temporal state, they typically require sequential processing, limiting scalability. TempoPFN tackles these issues by employing a GatedDeltaProduct architecture with state-weaving, enabling fully parallelizable training across various sequence lengths. This innovative design eliminates the need for techniques like windowing or summarization, ensuring robust temporal state-tracking while significantly improving efficiency.
The Power of Synthetic Data
A core aspect of TempoPFN’s success lies in its comprehensive synthetic data pipeline. This pipeline unifies a diverse array of data generators, including stochastic differential equations, Gaussian processes, and audio synthesis techniques. These generators are further enhanced with novel augmentations, creating a rich and varied training corpus that captures a wide spectrum of temporal patterns without relying on real-world data. This synthetic-only pre-training approach is crucial for ensuring reproducibility and preventing benchmark leakage, which can occur when models are inadvertently exposed to evaluation data during training.
The synthetic data generation process is elaborate, incorporating both existing and newly developed generators. For instance, adapted generators like ForecastPFN and KernelSynth create series with trends, seasonality, and noise. Novel generators, on the other hand, produce unique patterns such as ramp-like sawtooth functions, piecewise constant step functions, and anomaly-rich data with periodic or clustered spikes. The audio-inspired generators model complex real-world dynamics like stochastic rhythms, financial volatility, and network traffic bursts, while the stochastic differential equation (SDE) generator, based on a regime-switching Ornstein–Uhlenbeck process, introduces highly diverse temporal structures with regime shifts, non-stationarity, and periodicity.
Beyond generation, a sophisticated augmentation pipeline further enhances data diversity. This pipeline applies transformations in a structured sequence, including normalization, early-stage TS-Mixup (combining multiple series), and a core augmentation step that samples from categories like Invariances (time reversal, sign inversion), Structure (regime change, shock & recovery), Seasonality (amplitude modulation, calendar effects), Signal Processing (differential operators, integration), Discrete Effects (censoring, quantization), and Measurement Artifacts (resampling). This multi-faceted approach ensures that TempoPFN is exposed to a vast and varied set of time series characteristics during training.
Competitive Performance and Robustness
In zero-shot evaluations on the challenging Gift-Eval benchmark, TempoPFN demonstrated top-tier competitive performance. It outperformed all existing synthetic-only approaches and surpassed the vast majority of models trained on real-world data. Notably, TempoPFN achieved an overall CRPS (Continuous Ranked Probability Score) of 0.537, outperforming TabPFN-TS, the strongest synthetic-only baseline. Its point-forecast accuracy, measured by MASE (Mean Absolute Scaled Error), was also highly competitive at 0.797.
The model’s efficiency is another significant advantage, stemming from its fully parallelizable training and inference. Furthermore, TempoPFN exhibits superior robustness to missing values (NaNs) compared to other models like TiRex. While both models show performance degradation with increasing NaNs, TempoPFN’s error rate increased by only about 4% when 90% of the data was missing, whereas TiRex’s increased by over 11% under the same conditions.
The research paper highlights that the proposed SDE generator had the highest impact on performance, emphasizing the importance of exposing the model to time series with mean-reverting and noisy, continuous-time dynamics. The comprehensive augmentation pipeline also yielded consistent performance gains across all forecasting horizons.
Also Read:
- HiMAE: A Compact AI Framework for Smartwatches That Deciphers Wearable Health Data
- Spatiotemporal Error Adjustment Enhances Deep Learning Traffic Models
Future Directions
The authors have open-sourced their complete data generation pipeline and training code, providing a reproducible foundation for future research. While TempoPFN currently focuses on univariate time series, future work aims to extend the synthetic generation pipeline and state-weaving architecture to handle more complex multivariate scenarios. Additionally, incorporating pre-training on diverse real-world time series datasets could further enhance forecasting accuracy and generalization. The paper can be accessed here: TempoPFN Research Paper.
