TLDR: This research paper introduces a novel cross-pollination (XP) recommendation framework designed to connect grocery and general merchandise shopping. It uses a two-stage approach: first, Large Language Models (LLMs) generate novel cross-category product associations beyond historical data, and second, a transformer-based ranker leverages real-time shopping cart context to personalize and re-score these recommendations. The system demonstrated a 36% increase in add-to-cart rates with LLM-based retrieval and a 27% lift in ranking quality, significantly enhancing cross-category discovery on e-commerce platforms.
E-commerce platforms are constantly seeking ways to improve the shopping experience, and a new research paper from Walmart Global Tech introduces an innovative solution to a common challenge: recommending general merchandise to customers primarily focused on grocery shopping. Imagine buying milk and being subtly prompted to consider a milk frother, or purchasing pet food and seeing suggestions for pet toys. This is the essence of the new “cross-pollination” (XP) framework, designed to bridge the gap between distinct shopping categories and enhance customer discovery.
The paper, titled “Grocery to General Merchandise: A Cross-Pollination Recommender using LLMs and Real-Time Cart Context,” outlines a novel two-stage system. The first stage focuses on generating potential recommendations, while the second stage refines these suggestions based on a customer’s real-time shopping cart. The goal is to move beyond traditional recommendation methods that often struggle with category bias, offering more nuanced and contextually relevant suggestions.
Generating Smart Recommendations with LLMs
At the heart of the candidate generation stage are Large Language Models (LLMs). While historical purchase data provides valuable insights, it tends to reinforce existing shopping patterns. LLMs, like gpt-4o used in this research, are employed to identify entirely new, contextually relevant cross-category recommendations that might not emerge from past transactions alone. This involves a sophisticated three-stage process:
- Context Generation: An agent analyzes a grocery item (e.g., milk) and generates thematic contexts (e.g., “cereal,” “beverages,” “cooking”).
- Recommendation Generation: For each theme, another agent suggests specific non-grocery products (e.g., for “cereal,” it might suggest cereal bowls or milk pitchers).
- Contextual Evaluation: A final agent evaluates these suggestions based on their cross-category discovery potential, relevance, explanation quality, and purchase likelihood, filtering out low-scoring recommendations.
These LLM-generated recommendations are then matched to the product catalog using semantic search and further evaluated for quality. A dual evaluation framework combines an “LLM-As-Judge” score (for contextual relevance) and a “Cross-Encoder” score (for semantic alignment). Only recommendations that excel in both areas proceed, ensuring high-quality, relevant suggestions.
Real-Time Personalization with Cart Context
The second stage, the Cart XP ranker, addresses the dynamic nature of online grocery carts, which can contain many items and change frequently. This neural network-based system takes the initial cross-category candidates and re-scores them in real time, considering all items currently in the customer’s cart. This ensures that recommendations remain highly personalized and relevant as the shopping basket evolves. The ranker is trained on extensive cart interaction logs, optimizing for engagement signals like add-to-carts, and uses a transformer-based architecture to understand the joint semantics of all items in the cart.
Also Read:
- Enhancing Large Language Models for Better Sequential Recommendations
- Streamlining E-commerce Product Matching with Multi-Agent AI
Impressive Results and Future Outlook
The framework has shown significant practical effectiveness. Offline analysis and online A/B tests revealed a 36% increase in the add-to-cart rate for general merchandise items when LLM-based retrieval was used. The cart context-based ranker also achieved a 27% lift in NDCG@4, a common metric for ranking quality. Furthermore, LLM-based candidate generation led to a 4.7x increase in unique recommendations, vastly expanding the product discovery landscape.
While the system performs exceptionally well across most categories, the researchers identified persistent challenges with highly specialized products, particularly in Pet Supplies and certain Baby product recommendations. Future work aims to address these areas through more domain-specific fine-tuning, enhanced negative sampling, and advanced post-generation filtering.
This cross-pollination framework represents a significant step forward in e-commerce recommendations, offering a powerful way to enhance customer experience and drive cross-category discovery by intelligently leveraging LLMs and real-time shopping context. For more details, you can read the full research paper here.
