TLDR: Facebook has updated its Group Scoped Search by combining traditional keyword matching with advanced embedding-based retrieval (EBR) to improve relevance and diversity. This new hybrid system uses a fine-tuned ranking model and is evaluated by an innovative LLM-as-a-Judge framework, which uses large language models (like Llama 3) for scalable and consistent quality assessment. The results show significant improvements in user engagement and search quality.
Facebook, a social media giant, handles billions of search queries daily, helping users find everything from people and pages to groups and posts. Among these, “Group Scoped Search” allows users to look for information specifically within their social communities. Traditionally, this search relied solely on keywords, meaning it could only find posts that contained the exact words typed by the user. This approach often missed out on relevant content that used different, but semantically similar, terms. For instance, a search for “small individual cakes with frosting” wouldn’t find posts about “cupcakes” because the exact keyword “cupcakes” wasn’t in the query. This limitation led to fewer search results and potentially lower user engagement.
Introducing a more intelligent, semantic search capability presented several hurdles. First, using embedding-based retrieval (EBR), which understands the meaning behind words, adds significant computational demands and can increase search latency. Second, unlike keyword search where you can clearly see why a result matched, EBR works by comparing content based on its “meaning” in a high-dimensional space, making it harder to explain why certain results appear. For example, a search for “Italian coffee drink” might return “cappuccino” with a high similarity score, but without context, it’s not immediately obvious why. Finally, integrating this new semantic retrieval with existing keyword systems required complex updates to how results are ranked and combined.
To overcome these challenges, Meta has modernized Facebook Group Scoped Search by blending the best of both worlds: traditional keyword-based retrieval and advanced embedding-based retrieval. This hybrid approach aims to deliver more relevant and diverse search results, especially for natural language queries that don’t rely on exact keyword matches. The system’s architecture involves a parallel retrieval process. When a user types a query, it first undergoes preprocessing, preparing it for both retrieval methods. The keyword system uses Facebook’s “Unicorn” inverted index to find exact or close matches, while the embedding-based system, powered by a 12-layer, 200-million parameter model called SSR, converts the query into a dense vector. This vector is then used to find semantically similar posts in a precomputed “Faiss” vector index. This parallel processing ensures that the system benefits from both the precision of keywords and the semantic understanding of embeddings, all while maintaining high performance standards.
Once results are retrieved from both keyword and embedding systems, they are merged in a ranking stage. Here, a fine-tuned L2 ranking model, utilizing a Multi-Task Multi-Label (MTML) architecture, evaluates each candidate post. This advanced model considers both lexical features (like TF-IDF and BM25 scores from keyword matches) and semantic features (like cosine similarity scores from embedding matches). The ranking optimization focuses on three key user engagement signals: clicks, shares, and comments, balancing immediate relevance with long-term user interaction. This “plug-and-play” supermodel design allows for easier maintenance and faster updates, ensuring the system can quickly adapt and improve.
A crucial aspect of this modernization is the innovative evaluation framework. To continuously improve and maintain high relevance, Meta implemented an LLM-based evaluation system within its Build Verification Test (BVT) process. This “LLM-as-a-Judge” framework uses large language models, specifically Llama 3 with multimodal capabilities, to automatically assess the quality of search results at scale. The process involves replaying real search queries within group contexts and having the LLMs, guided by carefully designed prompts, deliver reliable relevance scores. This framework captures nuanced degrees of relevance, not just simple yes/no judgments, and evaluates performance across different result set sizes (e.g., top 5, top 10 results). This approach significantly accelerates development cycles by reducing the need for manual human labeling, minimizes variability and bias, and lowers evaluation costs, ensuring consistent and rapid quality assessment.
The evaluation results consistently showed improvements across all new components of the proposed system. The hybrid approach demonstrated complementary benefits from both embedding-based retrieval and the enhancements made to the ranking model. Metrics such as “relevant rate” (proportion of contextually relevant results) and “somewhat relevant rate” (including semantically related content) saw positive shifts, indicating a significant enhancement in user engagement and overall search quality. This work offers valuable insights for deploying and evaluating advanced retrieval systems in large-scale social platforms.
Also Read:
- Enhancing AI’s Understanding: A New Approach to Context-Aware Reasoning
- Improving RAG Performance with Hierarchical Document Chunking
For more technical details, you can refer to the original research paper: Modernizing Facebook Scoped Search: Keyword and Embedding Hybrid Retrieval with LLM Evaluation.
