TLDR: UniECS is a novel framework designed to unify multimodal e-commerce search, handling diverse visual and textual inputs seamlessly. It introduces a flexible architecture with a gated multimodal encoder that adaptively fuses information and manages missing modalities. Coupled with a comprehensive training strategy and a new benchmark (M-BEER), UniECS consistently outperforms existing models in various retrieval tasks, demonstrating superior performance and efficiency. Its successful deployment in a real-world e-commerce platform also shows significant improvements in key business metrics like Click-Through Rate and Revenue.
The world of e-commerce is constantly evolving, with a growing demand for sophisticated search systems that can understand and process diverse user inputs. Traditionally, online shopping platforms have relied on separate search methods for text (like keywords) and images (visual similarity). However, modern shoppers often combine these, for example, by uploading a photo of a jacket and adding a textual refinement like “in leather material.” This shift highlights a critical need for more flexible and unified search capabilities.
Current multimodal retrieval systems in e-commerce face two main challenges. Firstly, most systems are designed for very specific tasks with fixed input types, making them inflexible for varied shopping behaviors. Secondly, there’s a lack of comprehensive benchmarks to properly evaluate how well these unified search approaches perform across different combinations of images and text.
To address these limitations, researchers have introduced UniECS: a Unified Multimodal E-Commerce Search framework. This innovative system is designed to handle all possible retrieval scenarios, whether the input is an image, text, or a combination of both. The core idea behind UniECS is to create a single, flexible pipeline that can seamlessly process diverse inputs and retrieve relevant results.
How UniECS Works
UniECS features a flexible architecture centered around a novel “gated multimodal encoder.” This encoder is crucial because it can adaptively combine visual and textual information. It’s also smart enough to handle situations where one type of input might be missing, ensuring smooth operation without needing architectural changes. For example, if a user only provides an image, the system can still work efficiently.
The framework uses separate encoders for visual (image) and textual (text) inputs, which then feed into the gated multimodal encoder. This encoder employs a mechanism called “cross-modal attention” to allow visual and textual features to exchange information. An “adaptive gating mechanism” then determines how much of this cross-modal information to incorporate, effectively balancing the contributions from each modality. This prevents one modality from overpowering the other, especially when data is incomplete or noisy.
A Comprehensive Training Approach
To optimize its learning, UniECS employs a comprehensive training strategy that combines several specialized loss functions:
- Cross-Modal Alignment Loss (CMAL): This ensures that visual and textual representations of the same product are semantically consistent, making cross-modal searches more accurate.
- Cohesive Local Alignment Loss (CLAL): This helps the system distinguish between different items while keeping similar items grouped together, improving the quality of learned representations.
- Intra-Modal Contrastive Loss (IMCL): This enhances the discriminative power of single-modal representations, ensuring high-quality results even when only an image or only text is provided.
Additionally, UniECS uses an adaptive loss weighting scheme that dynamically adjusts the importance of each loss component during training. This ensures that all aspects of the model are optimized effectively, preventing any single loss from dominating the learning process.
Also Read:
- MOON: A New Generative AI Model for Deeper E-commerce Product Understanding
- PMTFR: A Novel Framework for Enhanced Composed Image Retrieval
New Benchmark and Impressive Results
To facilitate better evaluation of multimodal e-commerce search, the researchers also created M-BEER (Multimodal BEnchmark for E-Commerce Retrieval). This carefully curated dataset contains 50,000 product pairs, each with trigger text, trigger image, recall text, and recall image, allowing for standardized evaluation across nine distinct retrieval scenarios.
Extensive experiments demonstrate that UniECS consistently outperforms existing methods across M-BEER and other e-commerce benchmarks. Notably, UniECS achieves substantial improvements in cross-modal tasks, such as text-to-image retrieval (up to 28% gain in R@10 on M-BEER Bench compared to the best baseline). What’s even more impressive is that UniECS achieves these results with significantly fewer parameters (0.2 billion) compared to larger models like GME-Qwen2VL (2 billion) and MM-Embed (8 billion), making it much more efficient for real-world deployment.
Beyond experimental settings, UniECS has been deployed in the e-commerce search platform of Kuaishou Inc. for image search and same-style item retrieval. This real-world application has shown significant business improvements, including a 2.74% increase in Click-Through Rate and an 8.33% growth in total revenue. These results underscore the practical effectiveness of UniECS in enhancing user search experiences and boosting industry revenue.
For more technical details, you can read the full research paper: UniECS: Unified Multimodal E-Commerce Search Framework with Gated Cross-modal Fusion.
