TLDR: A new study introduces RePOPE-Spk, a benchmark designed to evaluate hallucinations in multimodal LLMs when queries are provided as speech under various acoustic conditions. The research reveals that spoken queries, especially those with environmental noise, significantly increase hallucination rates compared to text-based queries. While strategies like many-shot prompting and Chain-of-Thought reasoning offer some mitigation, they are insufficient to fully address the performance degradation. Both proprietary and open-source models exhibited similar vulnerabilities, indicating a systemic challenge in current MLLMs for robust voice-based multimodal interaction.
Recent advancements in artificial intelligence have brought us Multimodal Large Language Models (MLLMs) that can understand and process various types of information, including text, images, and audio. These models are becoming central to voice-driven interfaces in everything from mobile assistants to AR/VR platforms. However, a significant challenge remains: hallucinations. Hallucinations occur when an AI model generates information that is inconsistent with the provided context or factually incorrect. While hallucinations in vision-language models have been studied extensively, the impact of spoken queries on these models has largely been overlooked.
A new research paper titled “Evaluating Hallucinations in Multimodal LLMs with Spoken Queries Under Diverse Acoustic Conditions” by Hansol Park, Hoseong Ahn, Junwon Moon, Yejin Lee, and Kyuhong Shim addresses this critical gap. The researchers hypothesized that using spoken input instead of text would increase multimodal hallucinations, and that diverse acoustic conditions would make this problem even worse. To investigate this, they introduced RePOPE-Spk, an audio-augmented extension of the existing RePOPE benchmark. This new benchmark allows for the systematic evaluation of MLLMs when queries are provided as speech under various realistic noise and input conditions.
The RePOPE-Spk Benchmark
RePOPE-Spk builds upon the RePOPE benchmark, which is designed to measure object hallucination in vision-language models – instances where models claim objects are present in an image when they are not. For RePOPE-Spk, all text-based questions from the original benchmark are converted into spoken queries using a high-quality text-to-speech (TTS) model. This ensures a controlled evaluation by maintaining consistent speaker profiles and natural prosody. To simulate real-world scenarios, environmental noise from the ESC-50 dataset (including human, animal, interior, exterior, and natural sounds) is mixed with the spoken queries at different signal-to-noise ratios (SNRs), such as 0 dB and 5 dB.
Key Findings on Hallucination Escalation
The experiments, conducted on both proprietary models like Gemini and open-source models like Gemma, revealed several consistent and concerning trends:
- Spoken Queries Increase Hallucinations: Simply switching from text queries to clean audio queries led to a noticeable drop in accuracy and F1 scores across both models. This indicates that speech inherently introduces a performance challenge for MLLMs.
- Noise Exacerbates Hallucinations: As the noise levels increased (i.e., SNRs decreased from 5 dB to 0 dB), the models’ performance degraded substantially. This was particularly evident in precision scores, suggesting that models generate more false positives—hallucinating objects that aren’t there—under noisy conditions.
- Input Order Matters: While Gemini showed little sensitivity to the order in which images and spoken queries were presented, Gemma exhibited a significant performance drop when the spoken query came before the image. This suggests that some models might be less robust to variations in input modality sequences.
- Query Length Offers Partial Mitigation: Interestingly, extending the length of noisy spoken queries by padding the audio partially restored performance. However, even with longer queries, the models did not fully recover to the performance levels seen with clean spoken queries. This implies that more acoustic context can help, but it’s not a complete solution.
- Prompting Strategies Have Limited Impact: The researchers also explored mitigation strategies such as many-shot prompting (providing multiple examples) and Chain-of-Thought (CoT) reasoning. While increasing the number of examples initially showed gains, performance plateaued or even declined beyond a certain point, unlike in text-only LLMs. CoT strategies, especially those that are audio-aware (like instructing the model to transcribe the speech first), offered improvements but still did not close the performance gap with text-based queries.
A crucial observation was that both proprietary and open-source models exhibited similar vulnerabilities and relative performance drops under noise. This suggests that the limitations in handling spoken queries under diverse acoustic conditions are systemic to current MLLMs rather than specific to individual models.
Also Read:
- FATHOMS-RAG: A New Benchmark for Evaluating Multimodal RAG Systems
- AI Models Evaluate User Interfaces: A New Benchmark for Design Feedback
Conclusion and Future Directions
The findings from this research highlight a significant and underexplored challenge in building reliable voice interface systems. Spoken queries consistently degrade MLLM performance, and environmental noise and input order further amplify hallucinations. While current mitigation strategies offer some improvements, they are insufficient to bridge the gap to text-query performance. This underscores the intrinsic difficulty of the task and the urgent need for new approaches to achieve robust and reliable multimodal interaction in voice-driven AI systems. You can read the full paper for more details here: Evaluating Hallucinations in Multimodal LLMs with Spoken Queries Under Diverse Acoustic Conditions.
