TLDR: This paper introduces a novel method called “relative evaluation” for assessing anomalous sound detection (ASD) systems, especially when real anomaly data is scarce. It proposes using large language models (LLMs) to generate realistic, machine-type-specific synthetic anomalous sounds by interpreting fault descriptions and applying appropriate audio transformations. Experiments show that the performance rankings of ASD systems across different machine types are consistent between synthetic and real anomalies, validating the LLM-based approach as a scalable and reliable evaluation tool.
Detecting unusual sounds from machinery is crucial for predictive maintenance in industrial settings. These sounds can signal potential equipment failures, such as mechanical wear or misalignment, helping prevent costly downtime. However, a major challenge in this field is the scarcity of real-world anomalous sound data. It’s difficult and often hazardous to intentionally induce faults to collect this data, which limits both the training and evaluation of Anomalous Sound Detection (ASD) systems.
Traditional methods for generating synthetic anomalous sounds, like simple data augmentation or text-to-audio models, often fall short. They might produce unrealistic sounds or require existing anomalous samples for training, which defeats the purpose when such data is rare. This makes it hard to reliably benchmark ASD system performance across different machine types and fault conditions.
To address these limitations, a new concept called “relative evaluation” has been introduced. Unlike traditional absolute evaluation metrics (like AUC scores) which can fluctuate based on the severity of anomalies and become unreliable with scarce data, relative evaluation focuses on how an ASD system performs comparatively across different machine types. This helps maintenance engineers understand where a system is most trustworthy and where it might struggle, optimizing resource allocation and risk management in diverse industrial environments.
The paper proposes a novel method for synthesizing anomalous sounds to enable this relative evaluation. This approach leverages the common sense and contextual understanding of Large Language Models (LLMs). Instead of directly generating anomalies, the LLM interprets textual descriptions of machine faults and automatically selects appropriate audio transformation functions. This converts normal machine sounds into diverse and plausible anomalous sounds without needing prior anomalous data or manual intervention.
The process for generating these synthetic anomalies involves several steps. It starts with input metadata about machine types and operating conditions. An LLM (specifically, Flan-T5) then generates a descriptive caption. Normal machine audio is generated using a model like MIMII-Gen. This caption, along with a predefined library of anomalous sound effect functions (e.g., squeaking, rattling, grinding, humming, whistling, clicking, pulsing, popping, changes in tonal quality, or broadband noise increases), is fed to another LLM (GPT-4). The LLM analyzes the caption and intelligently selects the most suitable sound effect function. This selected function is then applied to the normal audio, transforming it into an anomalous sound with characteristics relevant to the described fault.
For evaluation, an unsupervised anomaly detection system, typically an autoencoder, is trained exclusively on real normal audio. This system calculates anomaly scores, and its performance is measured using AUC scores for each machine type. By comparing these AUC scores across different machine types, the relative strengths and weaknesses of the system can be identified.
Experiments were conducted using normal, synthetic, and real anomalous sounds from five machine types: bearings, gearboxes, fans, valves, and slide rails. The results showed a consistent ranking of detection difficulty across machine types for both synthetic and real anomalies. This indicates that the synthetic anomalies generated by the proposed LLM-based method are effective for evaluating the relative performance of ASD systems.
An ablation study further validated the LLM-based approach. It compared the GPT-4o-based method with a keyword-based manual mapping and a random selection of anomalies. Both the LLM-based and manual mapping approaches produced AUC-score rankings that closely aligned with those from real anomalies, while random selection showed no correlation. This highlights the critical role of contextual understanding, which LLMs possess, in generating realistic and relevant anomalies.
Also Read:
- Enhancing Spoken Language Correction with Automated Data Generation
- Assessing Voice Understanding in AI: Introducing the Speech Intelligence Quotient
In conclusion, this research introduces a significant advancement in evaluating UASD systems, especially in data-scarce scenarios. By introducing relative evaluation and an LLM-driven synthesis approach, it provides a scalable and reliable tool for benchmarking ASD performance and understanding detection difficulty across various machine types. For more details, you can refer to the full research paper: MIMII-Agent: Leveraging LLMs with Function Calling for Relative Evaluation of Anomalous Sound Detection.
