TLDR: DistDF is a novel time-series forecasting framework that addresses the bias in traditional likelihood-based methods (like MSE) caused by label autocorrelation. It achieves superior prediction accuracy by minimizing a newly proposed joint-distribution Wasserstein discrepancy, which effectively aligns the conditional distributions of model forecasts and actual labels. This method is theoretically justified, computationally efficient during training, and consistently improves the performance of various state-of-the-art forecast models without adding inference complexity.
Time-series forecasting, the art of predicting future values based on past observations, is a cornerstone of many modern applications. From predicting stock market trends and website traffic to guiding robotic movements, accurate forecasts are crucial. In the age of deep learning, developing effective forecast models relies on two key aspects: designing robust neural network architectures and crafting effective learning objectives to train these models.
While neural architectures have seen extensive research, with models like Transformers and linear models constantly evolving to capture complex data patterns, the design of learning objectives has remained comparatively less explored. Current standard approaches often rely on minimizing the conditional negative log-likelihood of the label sequence, typically estimated using the Mean Squared Error (MSE). However, this common practice has a significant flaw: it can be biased, especially when the label sequence itself has inherent autocorrelations (meaning future values depend on past ones).
This bias arises because MSE treats each future prediction as an independent task, overlooking the sequential dependencies within the data. Efforts to transform label sequences into ‘decorrelated’ components to eliminate this bias haven’t fully succeeded, as these transformations often only guarantee marginal decorrelation, not the conditional decorrelation truly needed to remove the bias.
Introducing DistDF: A New Approach to Time-Series Forecasting
To overcome this fundamental limitation, researchers have proposed a novel framework called DistDF, or Distribution-aware Direct Forecast. DistDF takes a different path, training forecast models by minimizing the discrepancy between the conditional distributions of the forecast and the actual label sequences. This means it aims to make the *entire distribution* of predicted future values match the *entire distribution* of actual future values, given the historical data.
Directly estimating these conditional discrepancies from limited time-series observations is challenging. To address this, DistDF introduces a new concept: the joint-distribution Wasserstein discrepancy. This clever mathematical tool provides a provable upper bound on the conditional discrepancy, meaning that by minimizing this joint discrepancy, DistDF effectively aligns the conditional distributions of interest. Crucially, this joint discrepancy can be reliably estimated from finite time-series observations, making it practical for gradient-based training methods.
How DistDF Works
The DistDF algorithm works by first generating a forecast sequence from the historical data. Then, it constructs two ‘joint sequences’ by concatenating the historical data with both the actual label sequence and the predicted forecast sequence. It then calculates the mean and covariance (which describe the distribution’s center and spread) for both of these joint sequences. The core of DistDF lies in using the Bruce-Wasserstein metric to quantify the discrepancy between these two joint distributions. This discrepancy term is then combined with the traditional MSE loss, allowing a balance between aligning the overall distributions and ensuring point-wise accuracy.
Also Read:
- OneCast: A Structured Approach to Cross-Domain Time Series Forecasting
- Time Series Learning Through Geometric Projections
Key Advantages and Performance
The theoretical foundation of DistDF is strong, with proofs demonstrating that minimizing the joint-distribution Wasserstein discrepancy to zero guarantees the alignment of conditional distributions. This leads to improved model training and, consequently, better forecast performance. Extensive experiments show that DistDF consistently enhances the performance of various state-of-the-art forecast models across diverse datasets. For example, on the ETTh1 dataset, DistDF reduced the MSE of the TimeBridge model by a notable margin.
Visual comparisons further highlight DistDF’s effectiveness. While traditional models might capture the general trend, DistDF produces forecasts that more precisely reflect subtle and rapid changes in the data, leading to more accurate real-world predictions. The framework is also model-agnostic, meaning it can be easily integrated as a plug-and-play component to improve the training of different forecast models, including Transformer-based and non-Transformer models.
The research also explored different components of the Wasserstein discrepancy, showing that aligning both the mean and covariance of the joint distributions contributes synergistically to the overall performance. Furthermore, DistDF demonstrated superior performance compared to other learning objectives and various discrepancy measures like Kullback–Leibler (KL) divergence and Maximum Mean Discrepancy (MMD).
In conclusion, DistDF offers a powerful and theoretically sound approach to time-series forecasting by directly addressing the limitations of biased likelihood estimation. By aligning conditional distributions through a novel joint-distribution Wasserstein discrepancy, it consistently improves forecast accuracy across a wide range of models and datasets. For more technical details, you can refer to the full research paper here.
