TLDR: A new research paper introduces a method to train AI vision models using only procedural (code-generated, non-realistic) data for perception, while storing real-world knowledge in a separate visual memory. This approach allows for perfect data unlearning, strong privacy guarantees, and efficient handling of sensitive information, all while maintaining competitive performance in tasks like visual similarity, classification, and segmentation. It offers a more transparent and controllable alternative to traditional models that embed knowledge directly into their weights.
Modern artificial intelligence systems, particularly those designed for visual tasks, learn by processing vast amounts of images and embedding that knowledge directly into their internal parameters, known as weights. While this method delivers impressive performance, it comes with significant challenges, including concerns about interpretability, privacy, and inherent biases. A major hurdle is the difficulty in adding or removing specific pieces of information once it’s been ‘baked’ into these weights, making data management and compliance with privacy regulations complex and costly.
To address these issues, a new research paper titled “Separating Knowledge and Perception with Procedural Data” proposes an innovative approach that fundamentally separates how a model ‘perceives’ from what ‘knowledge’ it stores. This is achieved by using a ‘visual memory’ – essentially an explicit database of reference image embeddings – to hold all real-world knowledge. Unlike traditional systems where knowledge is intertwined with the model’s core, this method allows for easy and efficient addition or removal of data from the memory, much like managing a database.
The key innovation lies in how the ‘perception’ component, specifically the embedding model that generates these visual features, is trained. Instead of using real-world images, the researchers train this model exclusively with ‘procedural data.’ This data is non-realistic and generated using simple code, such as OpenGL. By doing so, the embedding model learns useful visual features without ever being exposed to real-world knowledge, thus minimizing the risk of privacy leakage or bias from the training phase itself. This strong separation means that all real-world data is contained solely within the visual memory, providing perfect control over it.
The training process for the embedding model utilizes self-supervised learning, specifically the DINO framework, which helps the model learn robust visual representations from the abstract shapes and textures of procedural data. The paper introduces new procedural data generation processes, ‘Shaders KML’ and ‘Shaders KML Mixup,’ which have been shown to yield even stronger embeddings compared to prior methods.
Performance and Benefits
Despite being trained only on non-realistic data, the procedural model demonstrates surprisingly strong performance across various visual tasks when combined with visual memory. On the NIGHTS visual similarity benchmark, it performs within 1% of models trained on real-world data. For fine-grained classification tasks on datasets like CUB200 and Flowers102, the procedural model actually outperforms a model trained on the Places dataset by significant margins (8% and 15% respectively). It also remains competitive on larger datasets like ImageNet-1K, performing within 10% of the Places model.
Furthermore, the research highlights the model’s remarkable zero-shot and in-context semantic segmentation abilities, achieving results on par with models trained on real data for tasks like segmenting objects in COCO images. The approach also proves highly effective in handling sensitive data, such as facial images from CelebA or medical images from MedMNIST, by matching appearances and expressions without ever directly training on these sensitive real-world examples. This capability is crucial for applications in domains with strict data regulations.
A significant advantage of this approach is the ability to achieve ‘perfect unlearning’ and strong privacy guarantees. Since real-world data is only stored in the visual memory, removing a piece of data simply means deleting it from the memory, without the need for complex and costly retraining or fine-tuning of the core model weights. This makes it possible to provably evaluate the privacy impact of individual data samples.
Also Read:
- Boosting Offline Reinforcement Learning Generalization with Synthetic Data
- A New Path for Vision-Language AI: Text-to-Vision Mapping Without Alignment
Limitations and Future Outlook
While highly promising, the procedural models do have some limitations. Because they have never seen real-world objects during training, their representations can be excessively ‘local.’ This means that visually distinct parts of the same object (e.g., the spokes and center of a bicycle wheel) might be perceived as dissimilar, leading to occasional misclassifications or segmentation challenges where real-world models would recognize them as a unified entity. This gap in performance is an area for future research.
The paper also discusses the practical trade-offs. While the memory-based approach requires more storage for the embeddings, storage costs are relatively low compared to the significant computational resources (GPUs) needed for training traditional models. The training cost for the embedding model is also considerably lower than training a full parametric classifier, and inference can be made highly efficient with specialized libraries.
In conclusion, this research presents a compelling framework for building more transparent, controllable, and privacy-preserving AI vision models. By training perception on procedurally generated data and storing real-world knowledge in an explicit visual memory, the system offers unprecedented control over data, making it easier to manage, unlearn, and comply with privacy regulations, all while maintaining strong performance. You can read the full paper for more details at https://arxiv.org/pdf/2508.11697.
