TLDR: A new study, BehaviorLens, reveals that Multimodal Large Language Models (MLLMs) are significantly better at predicting a user’s next purchase when their past transaction data is presented as images (like scatter plots or flowcharts) rather than plain text. This visual approach can improve prediction accuracy by up to 87.5% without additional computational cost, suggesting a powerful new way to enhance personalized recommendation systems.
Multimodal Large Language Models (MLLMs) are rapidly changing how intelligent systems understand and react to sequential user behavior, such as shopping history. These advanced AI models are becoming central to applications like personalized recommendations and proactive customer support. However, a crucial question has remained largely unanswered: is it better for MLLMs to process user behavior data as text or as images?
A new research paper, titled “To See or To Read: User Behavior Reasoning in Multimodal LLMs,” introduces a systematic framework called BehaviorLens to explore this very question. Developed by Tianning Dong, Luyi Ma, Varun Vasudevan, Jason Cho, Sushant Kumar, and Kannan Achan from the Personalization Team at Walmart Global Tech, this study benchmarks six different MLLMs to assess the trade-offs between different data modalities.
Understanding User Behavior Through Different Lenses
The researchers investigated three distinct ways to represent user transaction data for MLLMs:
-
Textual Sequential Representation: This is the traditional method, where each purchase event (item, action, timestamp) is described in a simple, line-by-line text format. While it preserves granular detail, it can lose the overall structural information of a user’s journey.
-
Scatter-plot Representation: Here, the same purchase history is transformed into a scatter plot image. This visualizes temporal patterns, much like a time series graph, allowing the MLLM to potentially identify trends and recurring behaviors more easily.
-
Flow-chart Representation: Inspired by how visual compression can aid LLM reasoning, this method converts purchase history into a flowchart. Each purchase is a node, and connections illustrate temporal proximity, helping to convey the sequence and relationships between purchases.
The goal was to see which representation would lead to the most accurate next-purchase predictions and at what computational cost.
The Power of Visuals: Significant Accuracy Boost
The findings from BehaviorLens are quite striking. The study revealed that when user transaction data is presented as images – specifically scatter plots or flowcharts – MLLMs show a remarkable improvement in next-purchase prediction accuracy. In some cases, accuracy improved by as much as 87.5% compared to using an equivalent textual representation. This significant boost in performance was observed across various MLLMs, including larger models like GPT-4o and smaller ones like GPT-4.1-mini and Gemini-2.5-flash-lite.
Interestingly, this substantial improvement in accuracy often came without any additional computational cost in terms of token usage for GPT models, and similar token usage for flowchart inputs in Gemini models. While scatter plots sometimes required more tokens in Gemini models, the overall efficiency gain from improved accuracy is noteworthy. The study also looked at the quality of explanations provided by the MLLMs for their predictions. It found that the choice of input representation did not significantly alter the quality of these explanations, suggesting that the visual format primarily enhances the model’s ability to understand and predict, rather than just its ability to articulate its reasoning.
A Glimpse into AI Reasoning: Case Study
To illustrate the differences, the paper provides a case study using the Gemini-2.5-flash model. When given a user’s purchase history:
-
With text input, the model focused on purchase frequency and predicted ‘Cola’.
-
With flowchart input, it considered recency and complementary items, predicting ‘Pastries’.
-
However, with scatterplot input, the model was able to better capture recurring purchase patterns, leading to a prediction of ‘Granola Bars’, which was closer to the customer’s actual next purchase.
This example highlights how visual representations can help MLLMs identify patterns that might be less obvious in a flattened text format.
Also Read:
- Unpacking How AI Models Resolve Conflicting Information from Images and Text
- Enhancing Multimodal AI Understanding with Question-Guided Captions
Implications for the Future of Personalization
The research concludes that image representations offer a powerful way to enhance MLLM performance in reasoning over sequential user behavior data. This understanding is crucial for advancing personalization in agentic recommendation systems, which are designed to anticipate and respond to user needs proactively. Future work will focus on further optimizing these visual representations to capture even more complex temporal and spatial dynamics in user behavior, and on benchmarking these findings on larger datasets. For more details, you can read the full research paper here.
