TLDR: TimeRep is a novel anomaly detection method for time series data that uses intermediate layer representations from pre-trained Time Series Foundation Models (TSFMs) instead of final layers or task-specific heads. It builds a compressed memory bank of normal patterns and calculates anomaly scores based on the distance of new data’s representations to this bank. TimeRep also features a test-time adaptive memory bank to handle concept drift without retraining the model, demonstrating superior performance and efficiency on benchmark datasets.
Anomaly detection in time series data is crucial for ensuring the smooth operation of many real-world systems. Think about unusual vibrations in factory machines, slight delays in cloud services, or minor changes in health signals from wearable devices. These seemingly small issues can often be early warnings of major problems like system failures or medical emergencies. Therefore, being able to spot these anomalies quickly and accurately is vital for safety, reliability, and cost reduction across various industries, including manufacturing, healthcare, and IT.
Modern systems constantly generate vast amounts of time series data from sensors, logs, and transactions. This data is key to monitoring system behavior and identifying abnormal patterns. However, detecting anomalies is challenging because they are rare, diverse, and highly dependent on context. What might be an anomaly in one situation could be normal in another. Adding to this complexity is “concept drift,” where the definition of what’s “normal” changes over time due to seasonal trends, system updates, or evolving user behavior.
Traditional methods for anomaly detection, like rule-based systems or statistical models, often struggle to adapt to the complex and changing nature of real-world time series data. Deep learning models have shown more promise by learning intricate patterns, but they can be sensitive to shifts in data distribution and often need frequent retraining. This means many existing approaches fail to generalize well or adapt to new environments, leading to missed anomalies or false alarms, especially when data patterns change.
Recently, Time Series Foundation Models (TSFMs) have emerged as a powerful new approach for more generalizable anomaly detection. These models learn broad temporal patterns from huge and varied time series datasets, making them adaptable to many different tasks. However, most current TSFM-based methods typically use the representations from the model’s final layer, often combined with a specific “head” for tasks like reconstructing data or forecasting future values. This approach has limitations: final layer representations can sometimes overfit to the model’s initial training goal, and task-specific heads might not perform well if they try to reproduce anomalies too closely or struggle with unpredictable future patterns. Also, many of these methods lack a way to adapt to concept drift during real-time operation without updating the entire model.
Introducing TimeRep, a new unsupervised anomaly detection method that tackles these challenges. Instead of relying on the final layer, TimeRep directly uses representations from the intermediate layers of a TSFM. The core idea is that these intermediate layers capture more general and abstract features that are highly effective for spotting anomalies. TimeRep, developed by Chan Sik Han and Keon Myung Lee, offers a fresh perspective on how to leverage these powerful foundation models. You can read their full research paper here: Leveraging Intermediate Representations of Time Series Foundation Models for Anomaly Detection.
How TimeRep Works
During its training phase, TimeRep extracts these intermediate representations from normal time series data and stores them in a “memory bank.” To keep this memory bank efficient and prevent it from becoming too large or redundant, TimeRep uses a “core-set strategy.” This strategy intelligently selects a smaller, representative subset of these normal patterns, ensuring that the memory bank still covers the full range of normal behaviors without storing every single piece of data.
When new data comes in during inference (real-time operation), TimeRep calculates an “anomaly score” by measuring how different its intermediate representations are from the patterns stored in the memory bank. A larger distance means the new data is more likely to be an anomaly. TimeRep can use different ways to measure this distance, such as Euclidean distance, Mahalanobis distance, or a density-aware distance, with Euclidean distance often proving to be a strong default choice.
A key innovation in TimeRep is its “test-time adaptive memory bank.” This mechanism allows the system to adapt to concept drift without needing to retrain the entire foundation model. As new, non-anomalous data arrives, TimeRep selectively adds its intermediate representations to the memory bank if they are sufficiently “novel” (i.e., not redundant with existing patterns). This dynamic updating ensures that the memory bank stays current with evolving normal behaviors, maintaining high performance even as the underlying data patterns shift over time.
Performance and Key Findings
TimeRep was rigorously tested on the UCR Anomaly Archive, a comprehensive benchmark of 250 diverse univariate time series. The results show that TimeRep consistently outperforms a wide range of state-of-the-art methods, including traditional, deep learning, and other foundation model-based approaches. For instance, its center-aligned variant achieved a Top-1 accuracy of 77.6%, significantly higher than other methods.
One interesting finding was the importance of selecting the right intermediate layer. Experiments showed that representations from the 16th layer of the TSFM yielded the best performance, suggesting that these middle layers capture the most general and abstract features suitable for anomaly detection, unlike shallower layers (which capture low-level details) or final layers (which might be too specialized to the model’s pre-training task).
The position of the “reference time step” within a time series subsequence also mattered. Placing the reference at the temporal center of the subsequence (center-aligned) provided richer contextual information and led to better anomaly detection compared to placing it at the end (last-aligned).
Furthermore, the core-set compression strategy proved highly effective. Even with significant reductions in memory bank size (e.g., to 1,000 entries), TimeRep maintained strong accuracy while drastically cutting down inference time. This makes TimeRep highly practical for deployment in real-world systems.
The test-time adaptive memory bank also demonstrated its value, leading to consistent performance gains and enhancing robustness when dealing with concept drift. This means TimeRep can adapt to changing “normal” patterns without the costly and time-consuming process of retraining the entire model.
Also Read:
- GTHNA: A New Framework for Detecting Anomalies in Graph Data
- Dynamic Relational Priming: A New Approach to Transformer Attention for Time Series Forecasting
Conclusion
TimeRep represents a significant advancement in time series anomaly detection. By intelligently leveraging intermediate layer representations from time series foundation models and incorporating an adaptive memory bank, it offers a robust, efficient, and adaptable solution for identifying anomalies in dynamic real-world systems. Future work aims to extend TimeRep to multivariate time series and explore how different foundation model architectures might further enhance these intermediate representations for anomaly detection.
