TLDR: A new research paper introduces Approximate Domain Unlearning (ADU), a method for Vision-Language Models (VLMs) to selectively forget images from specific visual domains (e.g., illustrations) while preserving knowledge of other domains (e.g., real-world photos). This addresses the limitations of traditional ‘class unlearning’ by explicitly disentangling domain features and using instance-specific prompts, enabling more precise control over what AI models remember and forget, with potential applications in areas like autonomous driving.
Vision-Language Models (VLMs) have become incredibly powerful, capable of understanding and recognizing a vast array of objects across different visual styles. However, this broad capability often means they retain information that isn’t always necessary for specific tasks, leading to concerns about efficiency and even potential information leakage.
Traditionally, efforts to make AI models ‘forget’ have focused on what’s called ‘class unlearning.’ This involves retraining a model to no longer recognize specific object categories, like making a system forget what a ‘food item’ looks like. While useful, this approach has limitations in real-world applications.
Imagine an autonomous driving system. It needs to accurately identify ‘real cars’ on the road to ensure safety. But what if it encounters an advertisement depicting an ‘illustrated car’ on a billboard? If the system mistakes the illustration for a real vehicle, it could trigger dangerous, unintended actions. Simply forgetting the ‘car’ class entirely isn’t an option, as it still needs to recognize real cars.
This scenario highlights a critical gap that researchers from Tokyo University of Science, National University of Singapore, National Institute of Advanced Industrial Science and Technology (AIST), and University of Oxford have addressed with a novel concept: Approximate Domain Unlearning (ADU). ADU aims to teach VLMs to reduce their recognition accuracy for images from specified *domains* (like illustrations or paintings) while fully preserving their ability to recognize images from other, crucial domains (like real-world photographs).
The challenge with ADU is that pre-trained VLMs are designed for strong ‘domain generalization’ – they naturally see similarities across different visual styles. This means that features from various domains are often deeply intertwined within the model’s internal representations, making it difficult to selectively forget one domain without affecting others.
To overcome this, the researchers propose a two-pronged approach:
Domain Disentangling Loss (DDL)
This component explicitly works to separate the feature distributions of different domains in the model’s latent space. By making domains more distinct internally, the model can better differentiate between, say, a real car and an illustrated car. DDL uses a combination of cross-entropy and Maximum Mean Discrepancy (MMD) to achieve this separation, essentially pushing domain features further apart.
Also Read:
- Strategies for Skill Acquisition in Large Multimodal Models
- Balancing Data Deletion and Model Integrity in Federated Learning
Instance-wise Prompt Generator (InstaPG)
Domains themselves can be ambiguous. An ‘illustration’ can range from a highly realistic drawing to a simple cartoon. A single, fixed instruction for the model might not capture these subtle variations. InstaPG dynamically generates unique prompts for each individual image, allowing the model to adapt its understanding based on the specific visual characteristics of that instance. This fine-grained control helps the model to more accurately distinguish and forget specific styles within a broader domain.
Extensive experiments on several multi-domain image datasets, including ImageNet, Office-Home, Mini DomainNet, and DomainNet, demonstrated the effectiveness of this new approach. The ADU method significantly outperformed existing state-of-the-art VLM tuning techniques and class unlearning methods, showing superior performance in both forgetting unwanted domains and retaining desired ones.
The research also explored the robustness of ADU under various conditions, such as imbalanced datasets, partial domain-class overlap, and even scenarios with incomplete domain labels, showing promising results. This suggests that the method holds practical promise for real-world applications, including critical systems like autonomous driving where distinguishing between real and depicted objects is paramount.
This work introduces a new direction in machine unlearning, moving beyond class-level forgetting to a more nuanced, domain-specific control over what AI models remember. For more technical details, you can read the full research paper here.
