TLDR: The CHUG dataset is the first large-scale, crowdsourced collection of User-Generated High Dynamic Range (UGC-HDR) videos designed for quality assessment. It includes 856 source videos transcoded into 5,992 versions to simulate real-world streaming conditions, with over 211,000 subjective quality ratings collected via Amazon Mechanical Turk. This dataset addresses the gap in understanding quality degradations in diverse UGC-HDR content, providing a crucial benchmark for developing No-Reference HDR Video Quality Assessment models.
High Dynamic Range (HDR) videos are transforming how we experience visual content, offering brighter, more vibrant, and detailed images. However, the explosion of User-Generated Content (UGC) on platforms like YouTube and TikTok presents unique challenges for assessing the quality of these HDR videos. Unlike professionally produced content, UGC-HDR videos come with a wide array of capture conditions, editing styles, and compression artifacts, making it difficult to accurately measure their visual quality.
Existing datasets for HDR video quality assessment (VQA) primarily focus on professionally generated content, leaving a significant gap in understanding the real-world degradations found in UGC-HDR. To bridge this gap, researchers have introduced CHUG: a Crowdsourced User-Generated HDR Video Quality Dataset. This groundbreaking dataset is the first large-scale subjective study specifically designed for UGC-HDR quality.
What is CHUG?
CHUG comprises an impressive 856 unique UGC-HDR source videos. To simulate real-world streaming scenarios, these source videos were transcoded across various resolutions and bitrates, resulting in a massive collection of 5,992 videos. To gather perceptual quality ratings, a large-scale study was conducted using Amazon Mechanical Turk, collecting a staggering 211,848 individual ratings. This extensive data provides a crucial benchmark for analyzing the specific distortions that affect UGC-HDR videos.
The dataset is expected to significantly advance research in No-Reference (NR) HDR-VQA, which aims to assess video quality without needing an original, pristine reference video. Its large scale, diversity, and representation of real-world UGC make it an invaluable resource for the research community. The dataset is publicly available for researchers to explore and utilize.
How Was CHUG Created?
The creation of CHUG involved a meticulous process to ensure diversity and realism. Source videos were collected through an open call for UGC-HDR submissions from various personal devices, including iPhones, Samsung Galaxy, Google Pixel, and OnePlus smartphones. Strict filtering was applied to remove duplicates, objectionable content, and static videos, with each video trimmed to a maximum of 10 seconds.
To accurately mimic how social media platforms process videos, a bitrate ladder encoding strategy was applied. This means each video was encoded at multiple bitrates and resolutions (e.g., 360p at 0.2 Mbps, 1080p at 3.0 Mbps) to introduce realistic compression artifacts. This approach ensures that the dataset reflects the quality variations users experience when streaming content.
The Subjective Study
A large-scale subjective study was conducted on Amazon Mechanical Turk (AMT) to gather perceptual opinion scores. This was the first major AMT-based study for UGC-HDR, overcoming challenges related to device compatibility and display limitations. Over 700 subjects participated, providing an average of 35 ratings per video. Rigorous screening and quality control measures were implemented, including pre-screening for compatible devices, continuous HDR validation during the training phase, and post-study validation using control videos and ITU-R BT.500-14 filtering criteria. This ensured the reliability and high quality of the collected ratings.
Key Insights from the Data
The analysis of the collected scores revealed several interesting findings. The Mean Opinion Scores (MOS) were computed using the SUREAL method, a robust statistical approach that accounts for subject biases and inconsistencies. Some videos, particularly night scenes, those with extreme filters, or complex lighting, showed high MOS variance, indicating significant disagreement among raters. This highlights the challenging nature of real-world UGC-HDR content.
Researchers also found a positive correlation between spatial detail (textures, sharp edges) and perceived quality, though extremely high detail could make compression artifacts more noticeable. Moderate motion complexity was associated with higher quality, while excessive motion often led to lower ratings due to compression artifacts like blurring. Interestingly, video orientation (landscape vs. portrait) did not significantly influence perceptual quality, with both exhibiting similar MOS ranges.
As expected, MOS generally increased with higher bitrates and resolutions, confirming that better encoding preserves visual quality. However, the study also noted diminishing perceptual gains beyond certain resolution thresholds, influenced by content type and display scaling.
Also Read:
- OmniVideoBench: A New Benchmark for Advanced Audio-Visual AI Understanding
- Unlocking Video Understanding: A Deep Dive into Transfer Learning from Image-Language Models
CHUG’s Unique Contribution
Compared to existing HDR VQA datasets like LIVE-HDR and SFV+HDR, CHUG stands out with its broader MOS distribution, covering a wide range of low, medium, and high-quality HDR videos. This diversity is crucial for developing robust NR-VQA models that can accurately assess the varied distortions present in real-world UGC-HDR content. The dataset and scores will be publicly available upon publication at the CHUG project website.
In conclusion, CHUG represents a significant step forward in understanding and assessing the quality of user-generated HDR videos, providing a much-needed benchmark for the future of video quality research.
