TLDR: A new algorithm, Conformal Prediction for Time-series with Change points (CPTC), has been developed to improve uncertainty quantification in time-series forecasting, especially when data exhibits sudden shifts or ‘change points’. Unlike traditional methods that react to changes, CPTC anticipates these shifts by integrating a state prediction model with online conformal prediction. This approach ensures more reliable and adaptive prediction intervals, as validated by theoretical proofs and strong empirical results across various datasets, offering robust coverage and comparable prediction sharpness even with imperfect state predictions.
In the world of machine learning, predicting future events in time-series data is crucial for many applications, from forecasting electricity demand to predicting traffic patterns. However, these predictions become particularly challenging when the underlying processes generating the data suddenly shift, a phenomenon known as a ‘change point’. Imagine trying to predict electricity usage, knowing that demand patterns change drastically between day and night or weekdays and weekends. Traditional methods often struggle to provide reliable uncertainty estimates during these abrupt shifts.
A new research paper introduces an innovative algorithm called Conformal Prediction for Time-series with Change points (CPTC). This method aims to bridge a significant gap in current uncertainty quantification techniques for time series, especially those prone to sudden changes. The core idea behind CPTC is to integrate a model that predicts these underlying ‘states’ or ‘regimes’ with an online conformal prediction framework. This allows the system to anticipate and adapt to changes in data distribution, rather than just reacting to them after they occur.
Understanding the Challenge
Uncertainty Quantification (UQ) is vital for building trustworthy machine learning systems. Conformal Prediction (CP) is a popular method for UQ because it provides guarantees about prediction accuracy without making strong assumptions about the data’s underlying distribution. However, existing online CP algorithms for time series often react to distribution changes. This means they might under-cover (fail to include the true value in their prediction) during a shift and then over-cover later to compensate. This reactive behavior can be risky in practical applications where consistent, reliable predictions are needed.
The authors of CPTC observed that many real-world distribution shifts are, in fact, predictable. For instance, the ‘hidden dynamics’ of electricity demand are known to differ between day and night. This insight led them to leverage State Space Models (SSMs), particularly Switching Dynamical Systems (SDS), which are excellent at modeling time series where the system can switch between a discrete set of dynamics. SDS models can learn these different operating modes from data and even forecast future switching states.
How CPTC Works
CPTC operates by explicitly modeling these abrupt shifts. It uses an SDS model to predict the latent (hidden) state at any given time, indicating which of several possible dynamic regimes is currently active. This state prediction is then used to calibrate uncertainty for each regime separately. Essentially, CPTC runs multiple instances of online conformal inference, one for each potential underlying state, and then adaptively combines their confidence intervals as the system switches between modes.
The algorithm maintains a set of ‘nonconformity scores’ and an adaptive confidence level for each state. When a new data point arrives, CPTC generates state-specific prediction intervals based on the anticipated dynamics. These individual intervals are then aggregated into a single, final prediction interval. This modular approach means that the state model, the forecasting model, and the online adaptive conformal prediction components can all operate independently.
Key Advantages and Findings
The researchers proved that CPTC achieves valid coverage over time, even without strong assumptions about data stationarity or the perfect accuracy of state predictions. When the predicted state transitions align well with actual distribution shifts, CPTC can anticipate uncertainty and adapt much faster than other methods. This faster adaptation is crucial for real-world scenarios requiring quick responses to changing conditions.
Empirical results on six diverse datasets (three synthetic and three real-world, including electricity demand, traffic, and honey bee trajectories) demonstrated CPTC’s effectiveness. It consistently achieved more robust coverage, meaning its predictions were more reliably within the desired confidence level, compared to state-of-the-art baselines. While maintaining validity, CPTC also offered comparable prediction interval sharpness, meaning its uncertainty ranges were not excessively wide.
An interesting ablation study showed that even with imperfect state predictions, CPTC’s coverage performance remained strong, verifying the algorithm’s robustness. However, more accurate state predictions did lead to sharper (narrower) prediction intervals, indicating that while the algorithm is robust to errors, better state models can refine its precision.
Also Read:
- A New Metric for Evaluating Time Series Anomaly Detection: Confidence and Uncertainty Consistency
- Enhancing Safety Predictions for Complex Systems with Multi-Modal Behaviors
Looking Ahead
CPTC represents a significant step forward in providing reliable uncertainty quantification for complex, non-stationary time series data. Its ability to anticipate and adapt to change points makes it particularly valuable for applications where sudden shifts are common and accurate predictions are critical. While the current framework primarily applies to models with discrete states, future work could explore extending it to continuous states and further studying its implications for safety-critical applications requiring real-time adaptation.
For more in-depth technical details, you can read the full research paper here: Conformal Prediction for Time-series Forecasting with Change Points.
