TLDR: ADClick is an interactive image segmentation algorithm that efficiently generates pixel-level anomaly annotations for industrial products using a few user clicks and a text description. This significantly reduces the cost of data labeling and improves anomaly detection model performance. ADClick-Seg, a multi-modal extension, further enhances detection and localization by fusing visual and linguistic features, achieving state-of-the-art results in multi-class anomaly detection.
Industrial manufacturing relies heavily on automated inspection to ensure product quality. A common approach, Anomaly Detection (AD), typically trains on perfect, non-defective samples. While defective products are inevitably produced, using them to improve AD models has been challenging because it requires incredibly detailed, pixel-by-pixel annotations of defects, a process that is both time-consuming and expensive.
To overcome this significant bottleneck, researchers have introduced a novel approach called ADClick. This Interactive Image Segmentation (IIS) algorithm is designed to make the labeling of industrial anomalies much more efficient and precise. Instead of manual, pixel-level drawing, ADClick generates these detailed annotations from just a few user clicks and a short text description of the defect. This innovative method dramatically improves the performance of AD models, achieving impressive results, such as an AP of 96.1% on the MVTec AD dataset.
The core idea behind ADClick is to integrate multiple types of information: visual features from the image, linguistic prompts (text descriptions), and residual features (differences between a test image and a perfect reference). By combining these “multi-modal” cues, ADClick can create dense anomaly masks with a labeling cost comparable to much less precise “weak labels” like bounding boxes. For instance, annotating a scratch on a metal nut might only require 2-5 clicks and a brief text prompt.
Building on ADClick, the team also developed ADClick-Seg, a cross-modal framework that takes anomaly detection and localization a step further. ADClick-Seg aligns visual features with textual prompts using a prototype-based method. By merging pixel-level insights with language-guided information, ADClick-Seg achieves state-of-the-art results on complex “Multi-class” anomaly detection tasks, reaching an AP of 80.0%, PRO of 97.5%, and Pixel-AUROC of 99.1% on MVTec AD.
How ADClick Works
ADClick’s architecture is quite sophisticated. It takes four main inputs: the image being inspected, reference images of defect-free products, a language description of the defect, and the user’s clicks (indicating positive or negative areas). It uses “PosFAR” residual features, which are robust to noise, combined with click information and processed by a Swin Transformer. Defect-specific linguistic features guide this process through a mechanism called cross-attention. Simultaneously, a separate branch processes raw pixel information and clicks. The outputs from both branches are then merged and decoded to predict the detailed anomaly masks, which can be refined with additional clicks.
To generate the defect-specific linguistic features, the system uses large language models like ChatGPT. Users provide simple keywords, which ChatGPT then expands into descriptive phrases (e.g., “hazelnut displaying a visible shell perforation” from “hole defect on a hazelnut”). These phrases are then encoded into a compact linguistic representation using BERT, providing rich semantic cues that enhance detection quality.
Also Read:
- COGITAO: A New Framework for Advancing AI’s Compositional Reasoning
- Advancing GUI Understanding with SparkUI-Parser: A New Era for AI Interface Perception
ADClick-Seg for Enhanced Detection
ADClick-Seg refines the ADClick concept for fully automatic anomaly detection and localization. It simplifies certain modules to prevent overfitting, especially on smaller datasets. During testing, it can infer over various defect types, combining predicted maps to identify the final anomaly score. The research demonstrates that ADClick-Seg, particularly when combining residual, linguistic, and image features, outperforms many existing state-of-the-art anomaly detection methods.
A key finding is that labels generated by ADClick, even with just 3 or 5 clicks, provide information comparable to manually created ground-truth labels for training other anomaly detection models. This highlights ADClick’s potential to significantly reduce the cost and time associated with creating high-quality datasets for industrial inspection. For more technical details, you can refer to the full research paper here.
