TLDR: NILC is a novel framework for New Intent Discovery (NID) that combines embedding-based clustering with large language models (LLMs). It uses an iterative process featuring a ‘dual centroid scheme’ (Euclidean and LLM-generated semantic centroids) and ‘hard sample refinement’ where LLMs rewrite ambiguous utterances. For semi-supervised settings, it incorporates ‘seeding’ and ‘soft must-links’ from labeled data. NILC consistently outperforms state-of-the-art baselines in both unsupervised and semi-supervised settings across diverse datasets, demonstrating its effectiveness and robustness.
In the rapidly evolving world of artificial intelligence, understanding user intentions is paramount for effective dialogue systems, search engines, and personalized services. This challenge is particularly complex when users express entirely new or previously unseen intentions, a problem known as New Intent Discovery (NID). Traditional methods often struggle with this, leading to suboptimal performance and the need for extensive manual annotation.
Existing approaches to NID typically follow a two-step process: first, converting user utterances into numerical representations (embeddings), and then grouping these embeddings into clusters, which represent different intents. While straightforward, this ‘cascaded’ pipeline has a major drawback: the two steps don’t learn from each other. The quality of the initial embeddings heavily dictates the final clustering, and subtle meanings in text can be lost when relying solely on numerical distances between embeddings. Furthermore, while large language models (LLMs) offer immense potential, their direct use for NID can be computationally expensive and sometimes lead to less accurate results due to their general training.
Introducing NILC: A Smarter Approach to Intent Discovery
To overcome these limitations, researchers have developed NILC (New Intent Discovery LLM-assisted Clustering), a novel framework that intelligently combines the strengths of embedding-based clustering with the advanced understanding capabilities of large language models. NILC operates through an iterative process, where clustering assignments and text representations are continuously refined with LLM assistance, allowing both stages to mutually improve.
One of NILC’s core innovations is its ‘dual centroid scheme’. Instead of relying only on the average of numerical embeddings (Euclidean centroids) to define a cluster’s center, NILC also generates ‘semantic centroids’. These are textual summaries of each cluster’s theme, created by an LLM. By enriching the cluster representation with these human-readable summaries, NILC can capture the nuanced meanings that purely numerical embeddings might miss, leading to more accurate intent groupings.
Another key feature is ‘hard sample refinement’. In any clustering task, some user utterances are ambiguous, too brief, or contain jargon, making them difficult to assign confidently. NILC identifies these ‘hard samples’ and uses an LLM to rewrite them into clearer, more precise versions. This context-aware rewriting process helps to resolve ambiguity, and the refined utterances are then re-evaluated. This ensures that the clustering process is robust and less susceptible to noisy or unclear data.
Leveraging Labeled Data for Enhanced Performance
NILC also includes powerful optimizations for semi-supervised settings, where a small amount of labeled data is available. It uses ‘seeding’ to intelligently initialize clusters by aligning them with known intents from the labeled data. Additionally, ‘soft must-links’ are introduced, which act as gentle constraints during clustering, guiding ambiguous samples towards clusters that are semantically aligned with known intents. These techniques ensure that NILC can effectively leverage any available prior knowledge to boost its performance.
Also Read:
- How IDALC Boosts Intent Recognition in Dialog Systems
- AI Uncovers Hidden Subclasses in Time Series Data Using Saliency Maps and LLMs
Impressive Results Across Diverse Datasets
Extensive experiments conducted on six benchmark datasets, covering various domains from general queries to banking and technical questions, demonstrate NILC’s superior performance. It consistently outperforms a wide range of existing methods, including other LLM-based approaches, in both unsupervised (no labeled data) and semi-supervised settings. For instance, on the M-CID dataset, NILC showed significant improvements in accuracy, normalized mutual information, and adjusted rand score over the best baselines. The framework also proved robust to different choices of text encoders and large language models, highlighting its versatility and practical applicability.
NILC represents a significant step forward in New Intent Discovery, offering a powerful and flexible framework that harnesses the best of both embedding-based and LLM-driven approaches. By iteratively refining clusters and embeddings with LLM assistance, it provides a more accurate, interpretable, and cost-effective solution for understanding the ever-evolving landscape of user intentions. You can read the full technical report here.
