TLDR: A new study benchmarks multimodal large language models (MLLMs) like GPT-4o, Claude, and Llama against human perception in evaluating user interfaces (UIs). The research, titled “MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces,” investigates whether MLLMs can mimic human preferences for individual UIs and in pairwise comparisons across various cognitive, perceptual, and emotional factors. Findings indicate MLLMs can approximate human judgments, especially for visually concrete factors and when human preferences are clearly distinct, but they struggle with subjective emotional factors and fine-grained differences. The study positions MLLMs as valuable approximators for early-stage, low-resource UI evaluation, supplementing but not replacing human user research.
The world of user interface (UI) design is constantly evolving, and ensuring a positive user experience (UX) is paramount. Traditionally, this involves extensive user research, which can be time-consuming and resource-intensive, especially during the early stages of design exploration. A new study explores whether multimodal large language models (MLLMs) can step in as early evaluators, helping designers refine their options before formal human testing.
Unlike previous research that focused on user behavior in specific areas like e-commerce (tracking clicks or conversions), this paper delves into subjective user evaluations across a wide array of interfaces. The core question is whether MLLMs can mirror human preferences when assessing individual UIs and comparing them side-by-side.
Benchmarking MLLMs Against Human Judgments
Researchers from the University of California, Berkeley, and Adobe Research, including Reuben A Luera, Ryan Rossi, and Franck Dernoncourt, conducted a comprehensive benchmark. They used data from a crowdsourcing platform to evaluate leading MLLMs like GPT-4o, Claude 3.5 Sonnet, and Llama-3.2-11B-Vision-Instruct across 30 different user interfaces. The goal was to see how well these AI models aligned with human judgments on multiple UI factors.
How the Study Was Conducted
The study involved two main tasks. First, a diverse set of 30 UIs, including professionally made and some intentionally modified for quality variation, were collected across three categories: browsing & discovery, confirmation & feedback, and communication & engagement. These UIs were then judged based on nine factors, categorized into Cognitive (Ease of Use, Clarity, Visual Hierarchy), Perceptual (Memorable, Trust, Intuitive), and Emotional (Aesthetic Pleasure, Interest, Comfort).
Human participants provided 15,000 responses, evaluating each UI on a 7-point Likert scale for each factor and offering textual explanations. This extensive human dataset served as the ground truth for comparison.
Task 1: Absolute Score Prediction
In the first task, MLLMs were asked to predict absolute human scores for each UI and factor. The results showed that MLLMs generally approximated human preferences reasonably well, with their scores typically within 0.35 points of human scores on a 7-point scale. The models performed particularly well on cognitive and perceptual factors like ‘Trustworthy’ and ‘Visual Hierarchy’, likely because these factors have more concrete visual cues in the UI screenshots.
However, MLLMs struggled more with emotional factors such as ‘Interesting’ and some cognitive factors like ‘Ease of Use’. These are often harder for AI to quantify from visual cues alone, as they can be tied to a user’s technical literacy or subjective emotional response. While exact accuracy was moderate (Claude at 38%), the models showed a much higher ±1-accuracy (72-77%), meaning they were often within one point of the human score. Interestingly, MLLMs were quite effective at capturing the relative ranking of UIs, even if their absolute scores weren’t always perfect.
Task 2: Pairwise Preference Prediction
The second task involved MLLMs directly comparing two UIs and stating a preference. Here, GPT-4o and Claude performed best when judging which UI was more ‘Interesting’, achieving accuracies of around 75-78%. Conversely, they performed poorly on ‘Ease of Use’, with accuracies hovering around 50%, indicating near unpredictability.
A key finding from this task was that MLLMs’ accuracy significantly improved when there was a larger difference in human preferences between the two UIs being compared. When humans had very similar preferences, the MLLMs struggled. But when human preferences were clearly distinct, GPT-4o and Claude’s accuracy soared to 90-93%. Llama, however, remained closer to random across all factors in this task.
Also Read:
- AI Models Battle in Mortal Kombat II: A New Benchmark for Multimodal Intelligence
- Ro-Bench: A New Standard for Testing Video AI’s Resilience to Manipulated Content
MLLMs as Design Approximators, Not Replacements
The study concludes that MLLMs should be viewed as UI testing approximators rather than full replacements for human evaluation. Their moderate accuracy and reliability in generalized evaluations, especially when comparing UIs with clear disparities, make them valuable tools for early-stage user research. Designers can use MLLMs to get a general sense of direction during rapid iteration cycles, particularly when traditional research resources are limited.
However, the models’ struggles with fine-grained evaluations and factors that are difficult to infer from visual cues alone mean they are not precise enough for high-stakes validation testing at the end of the design process. The researchers emphasize that MLLMs should supplement existing human research, filling gaps where no user research would otherwise exist, rather than replacing it entirely.
This research also lays the groundwork for future advancements, suggesting that the benchmark data collected could be used to fine-tune MLLMs for better factor sensitivity or to explore more advanced prompting methods. The full research paper can be found here: MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces.
