TLDR: A new method called Quadratic Direct Forecast (QDF) improves multi-step time-series forecasting by using an adaptively updated quadratic-form weighting matrix during training. This approach simultaneously addresses two key issues: the correlation between future prediction steps (label autocorrelation) and the need for different importance levels for each forecasting task (heterogeneous task weights). Experiments show QDF consistently boosts the performance of various forecasting models.
The field of time-series forecasting, which involves predicting future values based on past observations, is fundamental to a wide array of applications, from meteorological predictions to financial stock market analysis and robotic trajectory planning. In the realm of deep learning, the development of robust forecasting models hinges on two critical components: the design of neural architectures and the formulation of suitable learning objectives for model training. While significant advancements have been made in neural architecture design, the area of learning objectives has remained relatively less explored.
A recent research paper introduces a novel approach called Quadratic Direct Forecast (QDF) that aims to significantly enhance the training of multi-step time-series forecasting models. The authors identified two primary issues with existing training objectives, such as the commonly used mean squared error (MSE):
Addressing Key Challenges in Forecasting
The first issue is the **label autocorrelation effect** among future steps. Traditional objectives often treat each future step as an independent task, overlooking the inherent dependencies and correlations that exist within the label sequence itself. This oversight can lead to a biased training objective.
The second challenge is the failure to set **heterogeneous task weights** for different forecasting tasks. Predicting each future step can be considered a distinct task, and these tasks often exhibit varying levels of difficulty and uncertainty. Assigning equal weights to all future steps, as many existing methods do, limits the potential forecasting performance.
To address these challenges, QDF proposes a novel quadratic-form weighted training objective. This objective utilizes a weighting matrix where the off-diagonal elements account for the label autocorrelation effect, while the non-uniform diagonal elements are designed to match the most preferable weights for forecasting tasks corresponding to varying future steps. This allows the model to learn both the relationships between future predictions and their individual importance.
How QDF Works
The QDF learning algorithm trains the forecast model using this adaptively updated quadratic-form weighting matrix. The process involves a sophisticated bilevel optimization problem. In an inner loop, the forecast model is trained using a fixed weighting matrix. In an outer loop, the weighting matrix itself is updated to improve the generalization performance of the trained model on a separate holdout dataset. This ensures that the learned weighting matrix is not just optimized for the training data but for how well the model performs on unseen data.
The workflow of QDF begins by initializing the weighting matrix as an identity matrix. The training dataset is then split chronologically into multiple non-overlapping subsets. The weighting matrix is iteratively refined by applying the atomic update procedure sequentially across these subsets until it converges or a predefined number of rounds is completed. Once the optimal weighting matrix is learned, it is then used to train the final forecast model by minimizing the corresponding negative log-likelihood objective. This approach is model-agnostic, meaning it can be integrated with various direct forecast models to improve their training and performance.
Also Read:
- Advanced Forecasting for Retail: How Temporal Fusion Transformers Improve Weekly Sales Predictions
- HIT-ROCKET: A New Era for Efficient Time Series Classification on Edge Devices
Empirical Validation and Impact
Comprehensive empirical evaluations were conducted on several public time-series forecasting datasets, including ETT, Electricity (ECL), Weather, and PEMS. The results consistently demonstrated that integrating QDF significantly improves the forecast accuracy of various state-of-the-art models. For example, on the PEMS08 dataset, QDF achieved a notable reduction in both Mean Squared Error (MSE) and Mean Absolute Error (MAE).
Qualitative comparisons further illustrated QDF’s effectiveness, showing that models trained with QDF were better at capturing subtle dynamics and periodic peaks in the data compared to models trained with traditional objectives. Ablation studies confirmed that both the heterogeneous task weights and the modeling of label autocorrelation effects contribute independently to performance gains, with their combined application yielding the best results.
QDF’s versatility was also highlighted, as it delivered consistent performance gains across different forecast models such as TQNet, PDF, FredFormer, and iTransformer. The research also explored the flexibility of QDF’s implementation, demonstrating that established meta-learning algorithms could be used to optimize the weighting matrix, further showcasing its adaptability.
This study marks a significant step forward in the design of learning objectives for time-series forecasting, offering a robust and effective solution to long-standing challenges. The full research paper can be accessed here: Quadratic Direct Forecast for Training Multi-Step Time-Series Forecast Models.
