TLDR: A new research paper introduces the Cross-Prompt Foundation Model (CPFM) for Black-Box Time-Series Domain Adaptation (BBTSDA). CPFM addresses the challenge of adapting AI models to new time-series data when only an API of the original model is available, ensuring privacy. It uses a pre-trained time-series foundation model with a ‘prompt tuning’ strategy, a dual-branch network, and reconstruction learning at both prompt and input levels. This innovative approach allows the model to learn distinct, complementary information and implicitly align with the target domain without needing labels. Experiments across human activity recognition, sleep stage classification, and machine fault diagnosis datasets show CPFM significantly outperforms existing methods, demonstrating its effectiveness in handling complex time-series data under black-box conditions.
In the rapidly evolving landscape of artificial intelligence, a significant challenge arises when models trained on one type of data need to perform well on a different, but related, type of data. This is known as the domain shift problem. While traditional methods often require access to the original training data or the full model, privacy and security concerns have led to the development of Black-Box Domain Adaptation (BBDA). In BBDA, only an Application Programming Interface (API) of the source model is available, meaning you can query it but not inspect its internal workings or access its training data.
While BBDA has gained traction, most existing solutions are designed for vision applications, like image recognition. These methods often fall short when applied to time-series data, which possesses unique ‘spatio-temporal’ characteristics – patterns that unfold over both time and space. Furthermore, the immense potential of ‘foundation models’ – large, pre-trained models capable of handling diverse tasks – has largely been unexplored in the context of black-box time-series domain adaptation (BBTSDA).
A recent research paper, “Black-Box Time-Series Domain Adaptation via Cross-Prompt Foundation Models”, introduces a novel approach called Cross-Prompt Foundation Model (CPFM) to tackle these very challenges. Authored by M. T. Furqon, Mahardhika Pratama, Igor ŠKRJANC, Lin Liu, Habibullah Habibullah, and Kutluyil Dogancay, CPFM leverages the power of foundation models and a clever ‘prompt tuning’ strategy to adapt to new time-series domains effectively, even when the source model remains a black box.
The Core Idea: Cross-Prompt Foundation Models (CPFM)
CPFM is built upon a time-series foundation model, specifically MOMENT, which is pre-trained on vast amounts of time-series data, giving it a strong understanding of general time-series patterns. Instead of retraining the entire large foundation model for a new task, CPFM uses ‘prompt tuning’. This means only small, task-specific parameters, called ‘prompts’, are adjusted, while the main backbone of the foundation model remains frozen. This significantly reduces the computational cost and the number of parameters that need to be learned.
A key innovation in CPFM is its ‘dual-branch network structure’. Imagine two parallel pathways within the model, each equipped with a unique prompt. These distinct prompts are designed to capture different, complementary aspects of the data distributions. The final prediction is then a combination of the outputs from both branches, ensuring a more robust and comprehensive understanding.
Reconstruction Learning for Better Adaptation
To further enhance domain adaptation, CPFM incorporates two types of reconstruction learning:
- Prompt Reconstruction: This strategy ensures that the prompts in each branch are truly distinct and generate complementary information. By trying to reconstruct the learned prompts, the model is encouraged to create unique prompts that avoid redundant information.
- Input Reconstruction: This method helps the model implicitly align with the target domain. It works by reconstructing samples from the target domain without needing their labels. This process allows the model to learn the underlying structure and patterns of the new domain, effectively performing adaptation without explicit guidance.
The overall learning process in CPFM combines these reconstruction losses with a standard cross-entropy loss, which is calculated using ‘pseudo-labels’ – labels generated by the source model. To mitigate the issue of noisy pseudo-labels (which are often inaccurate due to domain shifts), CPFM employs an Exponential Moving Average (EMA) rule, gradually replacing the source model’s predictions with the target model’s more reliable predictions over time.
Handling Multiple Source Domains
CPFM is also designed to handle scenarios with multiple source domains. In such cases, it aggregates predictions from multiple ‘teacher’ source models. It intelligently assigns ‘transferability weights’ to each source domain based on the uncertainty of its predictions (measured using Shannon entropy). Domains with lower prediction uncertainty are considered more similar and thus contribute more to the knowledge transfer. These weights are continuously updated using a moving average formula.
Impressive Experimental Results
The researchers rigorously validated CPFM’s advantages across three diverse time-series datasets: Human Activity Recognition (HAR), Sleep Stage Classification (SSC), and Machine Fault Diagnosis (MFD). In all experiments, CPFM consistently outperformed existing state-of-the-art black-box domain adaptation algorithms by significant margins. For instance, in the HAR dataset, CPFM showed an 11% gap to the second-best algorithm. Similar notable improvements were observed in the SSC and MFD datasets.
An ‘ablation study’ further confirmed the importance of each component of CPFM. Removing prompt tuning, input reconstruction, or prompt reconstruction individually led to a noticeable drop in performance, highlighting their crucial roles. The multi-source domain adaptation strategy also proved effective, with performance steadily improving as more source domains were utilized.
Also Read:
- Time Series Models Struggle with Real-World Video Data, Study Finds
- Time Series Analysis Enhanced by Joint Embedding Predictive Architectures
Conclusion
CPFM represents a significant step forward in black-box time-series domain adaptation. By combining the power of time-series foundation models with prompt tuning, a dual-branch network, and innovative reconstruction learning strategies, it effectively addresses the challenges of privacy-preserving adaptation for complex time-series data. The robust experimental results underscore its potential for real-world applications where data privacy and domain shifts are critical concerns. Future work aims to explore more complex scenarios, such as category shift problems in time-series domain adaptation.
