TLDR: FlowState is a new time series foundation model that uses a state space model encoder and a functional basis decoder to achieve sampling rate invariant and adaptable forecasting. It outperforms larger state-of-the-art models on benchmarks and generalizes well to unseen sampling rates, thanks to its continuous-time modeling and efficient parallel prediction training strategy.
In the rapidly evolving field of artificial intelligence, foundation models have revolutionized natural language processing, enabling powerful applications from text summarization to content generation. However, their success has not yet fully extended to time series forecasting, a critical area for predicting future trends in data like stock prices, energy consumption, or weather patterns.
Existing time series foundation models (TSFMs) often face significant challenges. They struggle to generalize across different lengths of historical data (context) and prediction horizons (target lengths). A major hurdle is their lack of adaptability to varying data sampling rates – for instance, predicting from hourly data versus daily data without extensive retraining. Furthermore, many current TSFMs are computationally inefficient, requiring vast amounts of data and processing power.
Addressing these limitations, researchers from IBM Research Europe – Zurich and ETH Zurich / UZH Zurich have introduced a groundbreaking new time series foundation model called FlowState. This innovative architecture is designed to inherently generalize across all possible temporal resolutions and dynamically adjust forecasting horizons, making it significantly more adaptable and efficient than its predecessors. You can read the full research paper here.
FlowState’s Core Innovations
FlowState distinguishes itself through two primary innovations:
A state space model (SSM) based encoder: Unlike many transformer-based models that struggle with time series data, FlowState leverages State Space Models. SSMs are stateful models, similar to recurrent neural networks, but with a key advantage: their linear state update allows for parallel processing, leading to improved computational efficiency. This encoder processes time series data as-is, without needing quantization or patching, and can naturally adjust to changes in input sampling rates.
A novel functional basis decoder (FBD): This decoder is a critical component that allows FlowState to produce continuous forecasts. Inspired by how SSMs interpret input sequences as coefficients of a polynomial basis, the FBD takes the encoder’s output and interprets it as coefficients of a functional basis (like Legendre polynomials). This enables the model to generate a continuous output function, which can then be sampled at any desired rate to produce forecasts of varying lengths without retraining. This is crucial for adapting to different forecasting needs and sampling rates.
Dynamic Adaptability and Efficient Training
A key feature of FlowState is its ability to dynamically adjust to the input sampling rate during inference. This is achieved by modifying a quantization parameter (∆) with an additional scaling factor. This ensures that the model can produce accurate forecasts even when encountering data with previously unseen sampling rates, a significant improvement over models that require training data across all possible scales.
FlowState also introduces an efficient pretraining strategy called “parallel predictions.” This scheme allows the model to concurrently train on a variety of context lengths by producing multiple forecasts in parallel from increasingly longer contexts. This not only significantly reduces training times but also enhances the model’s generalization capabilities and robustness to varying context lengths. To ensure strict causality and prevent information leakage during this parallel training, FlowState employs a causal normalization technique, using running mean and standard deviation instead of statistics from the entire context.
Performance and Robustness
Despite being a smaller model in terms of parameters (2.6M and 9.1M variants), FlowState has demonstrated state-of-the-art performance on two widely used benchmarks: GIFT-ZS and Chronos-ZS. It consistently outperforms much larger models, including TiRex, which was previously a benchmark leader. This highlights FlowState’s efficiency and scalability.
Furthermore, FlowState exhibits superior robustness to unseen sampling rates. Experiments on the Loop Seattle dataset, where data was subsampled to various intervals, showed FlowState consistently outperforming baselines across most frequencies, especially at uncommon intervals. This confirms its unique ability to generalize without requiring exposure to every possible frequency during training.
Ablation studies confirmed the effectiveness of FlowState’s individual components, showing significant performance drops when the time-scale adjustment or parallel prediction mechanisms were removed. The functional basis decoder also proved robust with different basis functions.
Also Read:
- Advancing Time Series Foundation Models with Synthetic Data Pretraining
- Dynamic Multi-Scale Coordination: Advancing Time Series Prediction
Conclusion
FlowState represents a significant leap forward in time series forecasting. By combining an SSM encoder with a novel functional basis decoder and an efficient parallel prediction training scheme, it offers unparalleled adaptability to varying sampling rates and forecasting lengths. Its ability to achieve state-of-the-art performance with a smaller model size makes it a promising foundation for future advancements in time series analysis.
