TLDR: Privacy Artifact ConnecTor (PACT) is an AI system that uses an embeddings-driven graph to link millions of diverse enterprise artifacts like code, data, and tools. Powered by a fine-tuned DRAGON embedding model, PACT helps AI agents understand complex relationships for privacy compliance, significantly improving information retrieval and search accuracy. Experiments show PACT enhances AI agent performance and acts as an efficient candidate fetcher for enterprise search.
In today’s fast-paced enterprise environments, organizations grapple with a vast and ever-growing collection of internal information, ranging from code repositories and data assets to various operational tools. Critical details needed for assessing privacy risks and ensuring regulatory compliance are often scattered across these diverse resources, each with its own unique way of being discovered and accessed. This complexity makes large-scale privacy compliance a significant challenge, as it requires understanding the intricate connections between these disparate pieces of information.
Addressing this challenge, researchers have introduced the Privacy Artifact ConnecTor, or PACT. PACT is an innovative, embeddings-driven graph system designed to link millions of artifacts across different types, generated by various teams and projects within an enterprise. At its core, PACT leverages the powerful DRAGON embedding model, fine-tuned with a specific learning approach to connect artifacts based on their textual components, such as metadata, ownership details, and compliance context. This allows PACT to create a unified, shared universe where the interconnected nature of diverse artifacts becomes clear.
How PACT Works: Building a Unified Understanding
PACT’s methodology involves several key steps to achieve its goal of creating a semantic understanding of enterprise artifacts:
First, it constructs a unified semantic graph by encoding each artifact into a shared embedding space. This means that regardless of whether an artifact is a code file path, an oncall team, or a product description, it is transformed into a numerical representation (an embedding) in a single space. The closer two embeddings are in this space, the more semantically similar their corresponding artifacts are. For instance, a code file and the team that owns it would be positioned close together.
Next, PACT fine-tunes the DRAGON embedding model using Meta’s internal data. This process sharpens and aligns the semantic relationships by leveraging known links between artifacts, such as a code file being owned by a specific team. The model learns to distinguish true connections from irrelevant ones, ensuring that artifacts connected through organizational knowledge, even if they don’t share explicit keywords, are clustered together in the embedding space. This step effectively customizes the general-purpose DRAGON model to the enterprise’s unique structure and terminology.
Using these fine-tuned embeddings, PACT builds a k-nearest-neighbor (KNN) graph, where each artifact is a node, and edges connect semantically related items. This graph serves as an intelligent index for semantic search. When a user enters a natural language query, PACT encodes it into an embedding vector and finds the most similar artifacts in its graph. Because all entity types share a single embedding space, the system can return a mix of relevant code paths, teams, and products. For example, a query like “Who’s the oncall for the payment service?” could retrieve the responsible team, related code repositories, and documentation.
Empowering AI Agents with PACT
One of PACT’s most significant applications is its ability to augment large language model (LLM)-based AI agents. These agents, operating in an iterative reasoning and acting loop, can use PACT as a semantic search tool. When an agent encounters an unknown term, needs to follow cross-references across different artifacts, or has to handle queries that exceed its immediate context length, it can invoke PACT. PACT retrieves the most relevant artifacts, providing the agent with domain-aware information that goes beyond simple keyword matching. This effectively gives the AI agent an extended, long-term memory of enterprise knowledge, enabling it to answer complex questions and perform sophisticated tasks more accurately.
Also Read:
- Enhancing AI Agents with Graph Structures: A Comprehensive Overview
- Enhancing AI Reliability: A Framework for Smarter Information Handling in Language Models
Experimental Validation: PACT’s Impact
Experiments have demonstrated PACT’s effectiveness across various scenarios:
- Fine-tuned Embeddings: When evaluating the task of mapping code files to oncall teams, PACT’s fine-tuned model significantly improved recall@1 from 18% to 53% compared to a baseline.
- Candidate Fetching: PACT was tested as a candidate fetcher in a recommender system for identifying enterprise product entities. A hybrid approach, combining PACT for initial candidate selection and an LLM for final classification, consistently outperformed LLM-only and KNN-only methods, achieving the highest accuracy (hitrate@1 from 25.7% to 44.9%) and reducing inference time by approximately 4x.
- AI Agent Enhancement: When integrated with an enterprise AI agent, PACT dramatically improved the agent’s ability to fetch relevant information. The average keyword match rate for artifact-related questions jumped from 9.6% for a base AI agent to 69.7% for an agent augmented with PACT.
In conclusion, PACT represents a significant step forward in managing and understanding the complex web of enterprise artifacts. By creating an embeddings-driven graph that links disparate information, PACT serves as a foundational element for large-scale privacy compliance, enhances enterprise search solutions, and empowers AI agents to reason more effectively over vast amounts of internal knowledge. For more detailed information, you can refer to the original research paper.
