TLDR: EcomEval is a new, comprehensive benchmark for evaluating Large Language Models (LLMs) in e-commerce. It addresses limitations of previous benchmarks by offering 37 diverse tasks across six categories, including multimodal tasks, using authentic customer data, and supporting seven languages (including low-resource Southeast Asian ones). The benchmark also incorporates difficulty levels and a semi-automatic answer generation pipeline with expert review. Evaluations of 19 LLMs reveal significant performance gaps, especially in multilingual and specific e-commerce tasks like generative content and user understanding, highlighting areas for future model improvement.
Large Language Models (LLMs) have shown impressive capabilities across many general language tasks, but their performance in specialized fields like e-commerce has remained less explored. Existing evaluation methods for e-commerce LLMs, such as EcomInstruct, ChineseEcomQA, eCeLLM, and Shopping MMLU, often fall short. They typically suffer from a limited variety of tasks, lack multimodal data (which combines text with images), rely on synthetic or curated information, and primarily focus on English and Chinese languages. This leaves e-commerce professionals without reliable tools to truly assess how well these advanced models handle the complex, real-world challenges of online shopping.
To address these critical gaps, researchers have introduced EcomEval, a groundbreaking benchmark designed for evaluating LLMs in e-commerce. EcomEval is comprehensive, multilingual, and multimodal, offering a robust testbed for understanding LLM performance in this dynamic domain. The benchmark covers an extensive range of scenarios, featuring six primary categories and 37 distinct tasks, including eight multimodal tasks that incorporate visual information. What makes EcomEval particularly valuable is its reliance on authentic data, primarily sourced from real customer queries and transaction logs. This ensures that the evaluation reflects the noisy and heterogeneous nature of actual business interactions, providing a more realistic assessment than benchmarks built on synthetic data.
Ensuring both the quality and scalability of reference answers was a key consideration in EcomEval’s development. The creators adopted a semi-automatic pipeline where large models initially draft candidate responses. These drafts are then meticulously reviewed and modified by a team of over 50 expert annotators, who possess strong e-commerce and multilingual expertise. This hybrid approach balances efficient production with rigorous quality control. Furthermore, EcomEval defines difficulty levels for each question and task category. This is achieved by averaging evaluation scores across models of varying sizes and capabilities, allowing for a challenge-oriented and fine-grained assessment of LLMs.
A significant advancement offered by EcomEval is its multilingual perspective. It spans seven languages, including English, Chinese, and five low-resource Southeast Asian languages (Indonesian, Vietnamese, Thai, Malay, and Portuguese). This broad language coverage is a crucial feature, as it addresses a major limitation of prior work that often neglected the global nature of e-commerce. The benchmark’s creators evaluated 19 open and proprietary LLMs on EcomEval, and the results revealed substantial performance disparities. These evaluations highlighted specific scenarios where general-purpose models struggled significantly within the e-commerce domain, underscoring the need for specialized development.
The findings from EcomEval provide actionable insights into the strengths and weaknesses of current LLMs in e-commerce. For instance, proprietary models like GPT and Gemini generally outperformed open-source models. GPT-5 and GPT-4.1 showed the best overall performance, though no single model dominated across all tasks. Interestingly, even models with impressive general NLP performance sometimes faltered on tasks requiring nuanced understanding of domain-specific terminology or multi-step reasoning based on product knowledge. Specifically, top-performing general models underperformed on complex tasks such as cross-lingual product question answering and conversational recommendations.
In terms of multilingual capabilities, all models performed better on English tasks compared to those in low-resource languages, with significant performance gaps observed. The benchmark also revealed that models struggled with e-commerce generative tasks, such as product tag and title generation, often missing key selling points. User understanding tasks, like query-product matching and search relevance, also proved challenging, with models frequently overlooking product categories and targeted buyers. Furthermore, tasks related to shopping guides and after-sales service, which are crucial real-world scenarios often absent from other benchmarks, showed considerable room for improvement for LLMs.
Also Read:
- Beyond Basic Q&A: ProfBench Challenges LLMs with Real-World Professional Expertise
- Decoding Global Trade: New Benchmark Reveals AI’s Struggle with Complex Customs Rules
Multimodal tasks, which involve analyzing both images and text, also presented unique challenges. While GPT-5 and GPT-4.1 led in this category, models often overestimated product similarity by focusing on titles and overlooking subtle visual differences in images. This highlights the importance of enhancing the multimodal capabilities of LLMs for fine-grained product understanding in e-commerce. By combining diversity, authenticity, quality, difficulty awareness, multilinguality, and multimodality, EcomEval establishes a rigorous and representative testbed. This will undoubtedly advance research and deployment of LLMs in e-commerce, guiding researchers and practitioners toward targeted improvements. For more details, you can read the full research paper here.
