TLDR: TokenCast is a new framework that uses large language models (LLMs) to improve time series forecasting by converting continuous numerical data into discrete ‘temporal tokens.’ This allows for seamless integration with unstructured textual information, leveraging the LLM’s understanding and generation capabilities. The framework involves discretizing time series, aligning these tokens with contextual text using a pre-trained LLM, and then fine-tuning the model to predict future tokens. Experiments on various real-world datasets show that TokenCast outperforms existing methods, demonstrating its effectiveness in handling complex, context-rich forecasting challenges.
Time series forecasting, the art of predicting future values based on historical data, is a cornerstone of decision-making across vital sectors like energy, healthcare, and finance. Despite significant advancements in this field, a persistent challenge remains: effectively integrating historical numerical sequences with rich, often unstructured, textual contextual features. Traditional forecasting models frequently struggle with this blend of different data types, limiting their accuracy and applicability in real-world scenarios.
Addressing this critical gap, researchers have introduced TokenCast, an innovative framework that leverages the power of large language models (LLMs) for context-aware time series forecasting. TokenCast proposes a novel approach by using language-based symbolic representations as a unified intermediary, allowing for a more cohesive understanding of both numerical and textual information.
How TokenCast Works: A Three-Stage Approach
The TokenCast framework operates through three distinct yet interconnected stages, designed to bridge the structural and semantic differences between continuous numerical data and discrete language tokens:
First, the Time Series Discretization stage transforms continuous numerical time series into a sequence of discrete “temporal tokens.” This is a crucial step, as it converts the numerical data into a format that structurally aligns with language tokens, making it compatible with LLMs. Unlike some irreversible discretization methods, TokenCast employs a dynamic and decoupled tokenizer that preserves temporal dependencies and allows for the reconstruction of the original time series from these tokens. This tokenizer includes a history-based reversible instance normalization layer, a causal temporal convolutional network (TCN) encoder, a vector quantization layer, and a Transformer-based decoder.
Next, the Cross-Modality Alignment stage tackles the semantic gap between the newly created temporal tokens and existing contextual features (like text). A pre-trained LLM is used to embed both types of tokens into a shared representation space. This is achieved by expanding the LLM’s original vocabulary to include the new temporal tokens and using a unified embedding matrix. Importantly, during this phase, the core LLM backbone remains frozen, with only the shared embedding matrix being updated through an autoregressive training objective. This ensures that the LLM’s vast pre-trained knowledge is leveraged while adapting it to understand the new time series vocabulary.
Finally, the Generative Fine-tuning stage adapts the aligned LLM specifically for forecasting. In this phase, the model is fine-tuned in a supervised manner to predict future temporal tokens. It learns to generate a structured response that can include both natural language analysis and the sequence of future time series tokens. During inference, the model receives historical data and contextual features, generates these future tokens, which are then decoded back into the original continuous numerical space using a frozen time series de-tokenizer. This process effectively transforms time series forecasting into a generative task, fully utilizing the LLM’s capabilities.
Also Read:
- TALON: A New Approach to Time Series Forecasting with Large Language Models
- Enhancing Language Models for Context-Aware Time Series Forecasting
Performance and Insights
Extensive experiments were conducted on a variety of real-world datasets, including economic indicators, health data, web traffic, and stock market fluctuations, all enriched with contextual features. TokenCast consistently demonstrated superior performance compared to several strong baselines, including other LLM-based models like Time-LLM and GPT4TS, as well as traditional Transformer-based and MLP-based methods. This highlights TokenCast’s effectiveness and generalizability across diverse domains.
Ablation studies confirmed the indispensable contributions of both the cross-modality alignment and generative fine-tuning stages to the framework’s overall performance. The research also revealed that incorporating any contextual features significantly improves forecasting accuracy, with “general information” (like domain knowledge) often yielding larger improvements than “local information” (event-specific details).
Interestingly, the study found that the optimal size for the codebook (which defines the number of discrete tokens) varies depending on the dataset, suggesting a balance between reconstruction fidelity and the complexity of the downstream forecasting task. Furthermore, smaller LLM backbones (e.g., Qwen2.5-0.5B-base) often outperformed larger ones, indicating that simply increasing model size doesn’t guarantee better performance for this specific application. The initialization method for the embedding layer also played a crucial role, with “mean initialization” proving to be the most robust.
The development of TokenCast marks a significant step forward in time series forecasting. By effectively converting continuous numerical data into a symbolic, language-like representation, it unlocks the full potential of pre-trained LLMs to understand, reason about, and generate future time series values in a context-aware manner. This unified token-based paradigm offers a robust and flexible solution for complex, real-world forecasting challenges. For more detailed information, you can refer to the original research paper.
