TLDR: TriSPrompt is a novel hierarchical soft prompt model designed to accurately detect rumors in multimodal social media data, even when modalities like images or comments are missing. It uses three types of prompts: Modality-Aware (MA) to recover missing information, Modality-Missing (MM) to adapt to incomplete data states, and Mutual-Views (MV) to leverage relationships between subjective (publisher) and objective (reviewer) perspectives. The model significantly outperforms existing methods and demonstrates strong robustness to high rates of missing data.
Social media platforms are bustling hubs for information exchange, where multimodal content – combining text, images, and comments – has become the norm. While this richness enhances engagement, it also creates a fertile ground for rumors, making accurate detection a significant challenge. A major hurdle in this task is the frequent occurrence of incomplete data, where parts of a post, like an image or a comment, might be missing.
Traditional methods for rumor detection often fall short in these real-world scenarios because they typically assume complete data during training. This limitation makes them ineffective when faced with the common problem of missing modalities in live social media feeds.
Introducing TriSPrompt: A Novel Approach
To tackle this critical issue, researchers have developed TriSPrompt, a hierarchical soft prompt model designed specifically for multimodal rumor detection, even when data is incomplete. TriSPrompt integrates three distinct types of ‘prompts’ – Modality-Aware (MA), Modality-Missing (MM), and Mutual-Views (MV) – to effectively process and detect rumors in these challenging conditions.
How TriSPrompt Works
The model operates through a clever three-pronged strategy:
1. Modality-Aware (MA) Prompt: This prompt is like a smart detective for missing information. It learns both the unique characteristics of each available modality (like text or comments) and the common features shared across them. When a modality is missing, the MA prompt uses the available information to reconstruct a representation of the missing part. This ensures that the model always works with a complete set of features, even if some were initially absent.
2. Modality-Missing (MM) Prompt: Beyond just recovering missing data, the MM prompt helps the model understand *that* a modality was missing and subsequently reconstructed. It acts as an indicator, enhancing the model’s awareness of the incomplete nature of the original data. This makes TriSPrompt more robust and adaptable to various missing information scenarios.
3. Mutual-Views (MV) Prompt: Rumors often involve different perspectives. The text and images in a post typically convey the subjective viewpoint of the person who published it. In contrast, comments often reflect the objective opinions of other reviewers or readers. The MV prompt is designed to uncover the hidden relationships between these subjective and objective perspectives. By differentiating and integrating these dual views, TriSPrompt can more accurately identify the tell-tale signs of a rumor.
Also Read:
- SafeCoDe: A Smarter Way for Multimodal AI to Handle Safety
- Disentangling Information for Smarter Recommendations
Performance and Robustness
Extensive experiments conducted on three real-world datasets (Weibo-19, Pheme, and Weibo-17) have shown that TriSPrompt significantly outperforms existing state-of-the-art methods. It achieved an accuracy gain of over 13% compared to other approaches. Crucially, TriSPrompt demonstrates exceptional robustness, maintaining strong performance even when a high percentage of modalities are missing. This is a vital advantage in real-world situations where data incompleteness is common and unpredictable.
The research highlights that while advanced Multimodal Large Language Models (MLLMs) are good at understanding multimodal content, they often struggle with rumor detection because they assume input information is factual and lack specific mechanisms to verify content accuracy. TriSPrompt, with its dedicated architecture, fills this gap effectively.
This innovative model represents a significant step forward in combating the spread of misinformation on social media, especially in scenarios where complete data is a luxury. The code and datasets for TriSPrompt are publicly available, encouraging further research and development in this critical area. You can find the full research paper here: TriSPrompt: A Hierarchical Soft Prompt Model for Multimodal Rumor Detection with Incomplete Modalities.
