TLDR: A new research paper introduces lossless compression as a novel and rigorous benchmark for evaluating time series models. Unlike traditional tasks such as forecasting or classification, which assess task-specific performance, lossless compression directly measures a model’s ability to capture the full generative distribution of data, grounded in Shannon’s source coding theorem. The open-source TSCom-Bench framework is proposed to standardize this evaluation, demonstrating that compression reveals distributional weaknesses in state-of-the-art models overlooked by classic benchmarks. The findings suggest lossless compression offers a more fundamental and stringent evaluation, with potential for unified pre-training strategies.
Evaluating how well time series models truly understand data has long been a challenge. Traditionally, these models are judged on tasks like forecasting future values, filling in missing data (imputation), spotting unusual patterns (anomaly detection), or categorizing sequences (classification). While these tasks have driven significant progress, a new study suggests they might not fully capture a model’s deep understanding of the data’s underlying structure.
Researchers Meng Wan, Benxi Tian, Jue Wang, Cui Hui, Ningming Nie, Tiantian Liu, Zongguo Wang, Cao Rongqiang, Peng Shi, and Yangang Wang introduce a novel approach: using lossless compression as a benchmark for evaluating time series models. This method is rooted in a fundamental principle of information theory, Shannon’s source coding theorem, which states that the most efficient way to compress data is directly related to how well you can predict its next elements. In simpler terms, a model that can compress data very well must have a profound grasp of its inherent patterns and statistical regularities.
Why Lossless Compression?
The core idea is elegant: if a model truly understands the generative process of a time series – how the data is created – it should be able to predict each subsequent data point with high accuracy. When these predictions are accurate, the data can be compressed into a much smaller size without losing any information. The length of the compressed data then becomes a direct, information-theoretic measure of the model’s capacity to capture the full distribution of the data.
This contrasts sharply with traditional tasks. For instance, forecasting models often aim to minimize errors like Mean Squared Error (MSE), which primarily focuses on getting the average prediction right. A model could achieve a low MSE even if it completely misunderstands the range or variability of the data. Similarly, classification might only require a model to identify a few key features, ignoring the vast majority of temporal dependencies. Imputation and anomaly detection also tend to focus on specific, localized aspects rather than the global statistical structure.
The researchers illustrate this with a clear example: a forecasting model might predict the mean perfectly for a data series that actually alternates between two distinct values. While its MSE would be optimal, its compression performance would be terrible because it assigns negligible probability to the actual values that occur. This highlights how compression provides a stricter and more comprehensive evaluation.
How the New Benchmark Works
The proposed evaluation architecture, called TSCom-Bench, involves three main steps. First, continuous time series data is converted into a discrete byte stream using a standardized encoding method (typically IEEE-754 32-bit format for numerical data). This ensures that the original data can be perfectly reconstructed. Second, a time series model is used to predict the probability distribution of the next byte, given all previous bytes. Finally, these probability predictions are fed into an arithmetic encoder. The better the model’s predictions, the more efficiently the arithmetic encoder can compress the byte stream, resulting in a smaller file size.
The TSCom-Bench framework is open-source, allowing researchers to easily integrate their time series models and evaluate them using standardized metrics like bits per byte (bpb), compression ratio (CR), and compression throughput (CT). You can find more details about the research and the framework here: Lossless Compression: A New Benchmark for Time Series Model Evaluation.
Also Read:
- New Study Challenges Foundation Forecasting Models’ Superiority Over Traditional Statistical Methods
- TimeMosaic: A New Approach to Time Series Forecasting with Adaptive Data Processing
Key Findings and Implications
Experiments on various real-world datasets, including traffic, electricity consumption, and weather data, showed that leading models like TimeXer, iTransformer, and PatchTST consistently performed well in compression. Interestingly, some models that didn’t always lead in forecasting tasks showed superior compression, indicating they capture richer data representations that traditional metrics overlook.
The benchmark also provided fascinating insights into the datasets themselves. For example, the Solar dataset exhibited exceptionally high compressibility because over half of its entries were zero, and it had a very limited number of unique values, making it highly predictable. Conversely, the PEMS08 dataset appeared near-incompressible because it was already stored in a compressed format, which the benchmark correctly identified as having minimal remaining redundancy.
Further validation using synthetic data with known properties confirmed that the compression approach accurately reflects a model’s ability to learn global statistical regularities and converges towards the theoretical information-theoretic limit as more data is processed. The research also demonstrated that time series models can outperform classical compressors even on mixed-modality data (audio, sensor readings, text), highlighting the practical utility of this approach for efficient data storage.
Comparing lossless compression with classic tasks revealed that while traditional tasks show no consistent correlation with each other, compression exhibits a moderate and uniform correlation with all of them. This suggests that compression measures a model’s fundamental understanding of the global data distribution, making it a powerful, unifying evaluation method. The authors propose that training models with compression-oriented objectives could serve as an excellent pre-training strategy, similar to how language models are pre-trained in natural language processing, before fine-tuning for specific downstream tasks.
In conclusion, lossless compression offers a rigorous, principled, and practical new benchmark for time series model evaluation. It uncovers distributional weaknesses that conventional tasks miss and provides a more holistic assessment of a model’s generative capabilities, paving the way for more robust and insightful time series AI.
