TLDR: This research paper highlights critical challenges in evaluating Time Series Foundation Models (TSFMs), which are powerful new forecasting tools inspired by LLMs. The authors identify issues like test set contamination (where test data leaks into training data), memorization of global patterns (models learning from widespread events like pandemics), and insufficient evaluation across time and diverse domains. They argue that current benchmarking methods are inadequate and propose 11 requirements for a new, robust evaluation framework, including the radical idea of continuously forecasting real future events to ensure true out-of-sample assessment and prevent an ‘evaluation crisis’ similar to that seen in LLMs.
Time Series Foundation Models (TSFMs) are an exciting new development in the world of forecasting, drawing inspiration from the success of Large Language Models (LLMs). These models promise a groundbreaking capability: zero-shot forecasting. This means they can predict future trends without needing to be specifically trained or fine-tuned for each new dataset, unlike traditional time series models. They achieve this by being pre-trained on vast amounts of generic and domain-specific time series data, allowing them to learn broad patterns and apply them to unseen data.
However, this powerful approach comes with significant challenges, particularly when it comes to evaluating how well these models actually perform. Just as with LLMs, the sheer scale of data used for pre-training TSFMs makes it incredibly difficult to ensure that the data used for testing hasn’t already been seen by the model during its training phase. This can lead to overly optimistic performance estimates and make fair comparisons between different TSFMs very difficult.
Key Evaluation Hurdles for TSFMs
The researchers identified several critical issues in current TSFM evaluation practices:
- Test Set Contamination: This is a major concern. It occurs when parts of the data intended for evaluating a model’s performance are inadvertently included in its pre-training data. The paper highlights instances where a dataset used for pre-training one model might be used as a test set for another, making direct comparisons impossible. For example, the “Australian Electricity Demand” dataset from the Monash collection has been used for pre-training, training/testing, and zero-shot forecasting across different models, rendering it unsuitable for fair benchmarking between them.
- Memorization of Global Patterns: Beyond direct data overlap, TSFMs can also ‘memorize’ global events. Imagine a model trained on various time series that all reflect the impact of a major event like a pandemic or an economic crisis. Even if the test data is technically separate, if it comes from the same period and is affected by the same global event, the model might appear to perform well simply by recalling these global patterns, rather than truly generalizing. This inflates performance metrics because the model has access to information that wouldn’t be available in a real-world prediction scenario.
- Limited Evaluation Dimensions: Traditional time series evaluation often involves cross-validation across different time periods (e.g., testing performance in summer vs. winter). However, current TSFM evaluations often use only a single time split per series, primarily to save computational resources. This means they might not fully assess a model’s consistency and robustness across various seasonal patterns or diverse domains.
- Lack of Domain Representativeness: TSFMs aim for generalizability across different domains (energy, transportation, weather, etc.). Yet, the datasets used for benchmarking are sometimes very specific and limited, such as the ETT dataset, which is based on measurements from just two transformers in China. It becomes questionable whether results from such narrow datasets can truly indicate a model’s general capability across a broader domain or different geographical regions.
Also Read:
- Rethinking Time Series Analysis: Integrating LLMs for Causal Reasoning and Explainability
- Understanding Transformer Limitations in Time Series Forecasting
Rethinking Benchmarking: A Call for New Standards
The paper argues that existing benchmarking strategies, while useful for traditional models, are insufficient for the unique challenges posed by TSFMs. Methods like time-based train/test splits, held-out datasets, competitions, synthetic data, and academic review processes each have limitations that prevent them from fully addressing information leakage and ensuring fair comparisons.
To address these issues, the authors propose 11 crucial requirements for a robust TSFM benchmark methodology. These include ensuring no test data is present in training, clear data split protocols, comprehensive evaluation across both time and domain dimensions, applicability to both existing and future models, flexibility in pre-training data, and mechanisms to mitigate global pattern memorization. Furthermore, the benchmark should be scalable, sustainable, representative of real-world data, support both zero-shot and fine-tuned evaluations, be inclusive, and robust against gaming or overfitting.
A particularly innovative idea put forth is a framework built around a continuously advancing global temporal split. In this vision, TSFMs would be challenged to forecast actual future events, with evaluations conducted retrospectively as real-world data becomes available. This dynamic approach inherently tackles information leakage because the ‘test set’ simply doesn’t exist during model training. While such a system would require significant development, it offers a promising path to resolve many current impasses in TSFM evaluation.
The research paper, available at arXiv:2510.13654, concludes by emphasizing that TSFMs are following a similar developmental path to LLMs, facing analogous evaluation challenges. Establishing robust, fair, and information-leakage-free benchmarking is essential to prevent an ‘evaluation crisis’ for TSFMs, ensuring meaningful progress in this rapidly evolving field.
