TLDR: This research introduces a new task called Complaint Description from Videos (CoD-V) to help users articulate detailed complaints by analyzing their uploaded product videos. It presents ComVID, a new dataset of 1,175 complaint videos with corresponding descriptions and emotional states, and proposes a multimodal VideoLLaMA2-7b model with Retrieval-Augmented Generation (RAG) that generates expressive complaint texts, even accounting for user emotions. This approach aims to improve e-commerce complaint resolution and assist users who struggle with written communication.
In the rapidly expanding world of e-commerce, where online shopping is becoming increasingly prevalent, especially in rural and underserved areas, understanding customer feedback is paramount. However, a significant challenge persists: users often struggle to articulate their complaints clearly and comprehensively through text alone. Imagine a user typing “worst product” for a broken headphone, when a short video could instantly show the specific defect, like a broken right earcup. This gap in communication often leaves issues unresolved and hinders effective customer service.
A new research paper, titled “When Words Can’t Capture It All: Towards Video-Based User Complaint Text Generation with Multimodal Video Complaint Dataset,” addresses this critical problem by introducing a novel task: Complaint Description from Videos (CoD-V). The goal of CoD-V is to empower users to express their grievances more effectively by generating detailed textual complaints directly from their uploaded videos. This initiative is particularly beneficial for individuals with expressive language disorders, limited literacy, or those who simply find it easier to demonstrate an issue visually rather than describe it in writing.
Introducing ComVID: A New Dataset for Video Complaints
To facilitate this new research direction, the authors have created ComVID, a unique video complaint dataset. ComVID comprises 1,175 complaint videos, primarily focusing on electronic products (like keyboards, earbuds, mice, headphones) and other items (bags, shoes, household goods) from Amazon reviews. Each video in the dataset is meticulously annotated with a corresponding detailed description of the complaint and, crucially, the emotional state of the complainer (e.g., dissatisfaction, blame, frustration, disappointment). This rich dataset provides an invaluable resource for training and evaluating models that aim to understand and articulate video-based complaints.
The dataset collection involved scraping 1- and 2-star Amazon reviews, transcoding video URLs, and ensuring a balanced distribution of complaint aspects across various domains like Fashion, Electronics, Household, and Others. Key issues covered include quality, functionality, defects, missing items, refunds, and performance. A team of expert linguists manually annotated the videos, ensuring high-quality descriptions and accurate emotional labels, with a robust inter-annotator agreement.
The Multimodal VideoLLaMA2-7b Model for Complaint Generation
To tackle the CoD-V task, the researchers propose a sophisticated multimodal Retrieval-Augmented Generation (RAG) embedded VideoLLaMA2-7b model. This model is designed to process both the visual information from the complaint video and a textual prompt, along with the user’s associated emotional state, to generate a coherent and contextually accurate descriptive complaint. The architecture involves two main steps:
First, a Multimodal Retrieval (MR) framework leverages a large dataset of Amazon product reviews (text and images) to extract multimodal embeddings. When a new complaint video is provided, keyframes are extracted and combined with product aspects to form a query. This query is then used to retrieve the most similar existing complaint reviews from the extensive Amazon dataset, providing a rich, context-augmented input.
Second, the VideoLLaMA2-7b model is fine-tuned on supervised video-text pairs. During the complaint generation process, the model takes the enriched input (video features, textual prompt, user’s emotional state, and retrieved complaint reviews) to produce a refined and detailed complaint description. The inclusion of the user’s emotional state is a critical component, significantly enhancing the contextual relevance and tone of the generated complaint.
Distinguishing CoD-V from Traditional Tasks
The paper also introduces a new evaluation metric called Complaint Retention (CR) to specifically assess how well the generated output captures the nature of the complaint. This metric considers sentiment score, emotion detection, and aspect identification, differentiating CoD-V from standard video summarization or description tasks that often produce generic outputs. The research demonstrates that the proposed VideoLLaMA2-7b+MR model consistently outperforms conventional baselines and state-of-the-art methods across various evaluation metrics, especially when incorporating emotional cues.
Also Read:
- SafeCoDe: A Smarter Way for Multimodal AI to Handle Safety
- AHA: Real-Time Highlight Detection for Streaming Video Without Future Context
Societal Impact and Future Applications
The implications of this research are far-reaching. By providing a platform for users to express complaints through video, it can significantly improve customer service responsiveness in e-commerce. Less literate users, or those with communication difficulties, can now convey nuanced issues more effectively, leading to faster and more accurate resolutions. Beyond e-commerce, this video-to-text generation capability could be applied to other domains, such as real-time patient issue reporting in healthcare, automated summarization of movie trailers, or even toxicity detection in online content from multimodal inputs.
This groundbreaking work lays the foundation for a new research direction, offering a powerful tool to bridge communication gaps and enhance user experience across various platforms. For more details, you can refer to the full research paper here.
