TLDR: Time Series Foundation Models (TSFMs) often fail to generalize effectively in real-world industrial applications, despite strong performance on public benchmarks. This research identifies “spectral shift”—a mismatch in dominant frequency components between pretraining data and downstream tasks—as a primary cause. Through an industrial case study in mobile gaming and controlled synthetic experiments, the authors demonstrate that TSFMs underperform significantly when encountering frequency bands not represented during their pretraining, highlighting the critical need for frequency awareness in TSFM development and deployment.
Time Series Foundation Models (TSFMs) have recently shown impressive capabilities on various public benchmarks, leading many to believe they could bring a transformative ‘BERT moment’ to the world of time series analysis. However, their true effectiveness in real-world industrial applications has remained a significant question.
A new research paper, titled “Frequency Matters: When Time Series Foundation Models Fail Under Spectral Shift”, delves into why these advanced models often struggle to generalize beyond their training environments. The authors, Tianze Wang, Sofiane Ennadir, John Pertoft, Gabriela Zarzar Gandler, Lele Cao, Zineb Senane, Styliani Katsarou, Sahar Asadi, Axel Karlsson, and Oleg Smirnov, propose that a key factor is ‘spectral shift’ – essentially, a mismatch between the dominant frequency components in new tasks and those the models were exposed to during their initial training.
The research provides compelling evidence from an industrial-scale player engagement prediction task within mobile gaming. In this real-world scenario, TSFMs surprisingly underperformed compared to more traditional, domain-adapted baseline models. To understand this mechanism better, the researchers designed controlled synthetic experiments. They contrasted signals with frequency bands that the models had ‘seen’ during pretraining versus those they had ‘unseen’. The results consistently showed a systematic degradation in performance when the models encountered these spectral mismatches.
The Challenge in Real-World Gaming
The industrial case study focused on predicting player engagement in a popular mobile game, Candy Crush Saga. The task involved predicting a player’s future behavior, such as making a purchase or their overall engagement score, over a 30-day horizon, based on their past gameplay history. The input data consisted of multivariate time series, capturing up to 512 completed game rounds and including 32 different features like progression, gameplay, resource usage, and contextual information.
When evaluated on this complex dataset, TSFMs like MOMENT-small, despite their public benchmark success, lagged significantly behind models like PatchTST and even simpler tabular baselines such as XGBoost and TabNet. This stark difference highlighted a gap between theoretical performance and practical utility.
Unpacking the Frequency Perspective
The researchers hypothesized that TSFMs might be relying on specific frequency components memorized during pretraining. If a new dataset’s dominant frequencies fall outside this learned spectrum, the model’s ability to generalize suffers. A spectral analysis of the gaming dataset confirmed this: it exhibited fundamentally different dominant frequencies compared to the datasets used to pretrain MOMENT, such as FordA and FaultDetectionA from public repositories.
To rigorously test this hypothesis, the team created synthetic datasets. They extracted dominant frequencies from pretraining data to define ‘seen’ frequency bands. Then, they synthesized new signals: one set with frequencies sampled from these ‘seen’ bands and another set with frequencies sampled from ‘unseen’ bands, shifted to be disjoint from the seen ones. They then froze the pretrained TSFM backbone (MOMENT-small) and trained lightweight regression or classification heads on these synthetic variants.
The experimental results were clear: for both regression and classification tasks, the TSFM consistently performed better on data with ‘seen’ frequency bands than on data with ‘unseen’ bands. For instance, in regression tasks, the Mean Squared Error (MSE) and Mean Absolute Error (MAE) were notably higher for unseen datasets across various benchmarks like ElectricDevices and FaultDetectionB. Similar trends were observed in classification accuracy and AUC.
Also Read:
- Navigating Spatio-Temporal Forecasting: When Graph Neural Networks and Foundation Models Shine
- FreqRec: Enhancing Sequential Recommendations with Dual-Path Frequency Analysis
Implications for the Future of TSFMs
While the study acknowledges limitations, such as focusing on one industrial domain and a single TSFM configuration, the consistent findings offer crucial practical guidance. The authors recommend that practitioners:
- Quantify the spectral overlap between their downstream data and the pretraining datasets used for TSFMs.
- Incorporate frequency-aware data augmentations or light adaptation techniques when spectral overlap is low.
- Adopt new benchmarks that explicitly test the robustness of TSFMs under various spectral shifts.
In conclusion, this research underscores that for Time Series Foundation Models to truly achieve universality and move beyond benchmark success, addressing the critical issue of spectral shift is paramount. It motivates new pretraining and evaluation protocols that explicitly account for the diverse frequency compositions found in real-world time series data. You can read the full paper here.
