TLDR: A new large-scale dataset called Kaputt has been introduced to benchmark visual defect detection in complex retail logistics environments. It features over 230,000 images, 29,000+ defective instances, and 48,000+ unique items, significantly larger and more diverse than previous datasets. Initial evaluations show that state-of-the-art anomaly detection methods struggle, achieving less than 57% AUROC, highlighting the need for new, more robust solutions for real-world retail challenges.
Automated visual defect detection is a crucial component of quality assurance in industrial and logistics operations. However, while manufacturing scenarios often involve highly controlled environments and a limited number of object types, the world of retail logistics presents a far more complex challenge. A new research paper introduces ‘Kaputt: A Large-Scale Dataset for Visual Defect Detection’, aiming to bridge this gap and push the boundaries of current anomaly detection methods.
The paper, authored by Sebastian Höfer, Dorian F. Henning, Artemij Amiranashvili, Douglas Morrison, Mariliza Tzes, Ingmar Posner, Marc Matvienko, Alessandro Rennola, and Anton Milan, all from Amazon (with Ingmar Posner also affiliated with the University of Oxford), highlights the limitations of existing benchmarks like MVTec-AD and VisA. These datasets, primarily focused on manufacturing, have seen state-of-the-art methods achieve near-perfect scores, indicating a saturation point.
The Unique Challenges of Retail Logistics
In contrast to manufacturing, retail logistics deals with an immense diversity of items, ranging from consumables to electronics, each with unique physical properties. Products are often observed in varied poses and orientations, and defects can be subtle, from minor creases to significant spillages. The rarity of defects also makes collecting training data expensive and time-consuming, leading to highly imbalanced datasets that necessitate unsupervised or anomaly detection (AD) approaches.
The Kaputt dataset directly addresses these challenges. It is an unprecedented resource, boasting over 230,000 images and more than 29,000 defective instances, making it 40 times larger than MVTec-AD. It features over 48,000 distinct objects, presented in random poses and orientations, closely mirroring real-world retail logistics scenarios.
Dataset Structure and Annotation
The dataset is structured into ‘query’ and ‘reference’ sets. Query images come with detailed annotations, including a unique item identifier, defect severity (no defect, minor, major), specific defect types (such as penetration, deformation, actuation, deconstruction, spillage, superficial, or missing unit), item material, and segmentation masks. Each query image is also associated with up to three unannotated reference images, typically depicting the item in a ‘normal’ condition.
Data collection involved a two-stage strategy: initially, human operators flagged defective items, followed by an iterative mining process using a binary classifier to identify potential defect candidates. This approach helped overcome the rarity of defective samples. Annotations were meticulously performed by three annotators, with majority voting used for aggregation, ensuring high-quality labels for defect severity and type.
Benchmarking State-of-the-Art Methods
To validate the difficulty of the Kaputt dataset, the researchers conducted an extensive evaluation of multiple state-of-the-art anomaly detection methods. The results were striking: leading methods failed to surpass 56.96% AUROC (Area Under the Receiver Operating Characteristic curve) on this new dataset. This stark contrast to the near-perfect scores on older benchmarks underscores the significant challenge posed by the variability in item pose and appearance within retail logistics.
Supervised baselines, when given access to a large training set of defective instances, performed better, achieving up to 94.27% AUROC. However, they still struggled with subtle anomalies, rare defect types, and ‘adversarial’ items (e.g., items with damage-like designs). Interestingly, methods that naively combined training data with reference images sometimes performed worse than reference-free approaches, suggesting that simply adding reference images without sophisticated integration can hinder performance.
Also Read:
- Uncovering Vision-Language Models’ Struggle with Counting Multiple Objects
- Syn-Diag: Smart Fault Diagnosis for Industry with Cloud-Edge AI
Future Directions
The Kaputt dataset sets a new benchmark for visual defect detection in retail logistics. It encourages future research to develop more robust and generalizable methods that can handle significant item and pose variability, effectively leverage both training data and reference images, and even explain their reasoning for defect detection. The dataset is publicly available for download at www.kaputt-dataset.com.
