TLDR: This research paper investigates the effectiveness of 18 semantic similarity measures across natural language and code. It reveals that traditional and embedding-based methods often fail catastrophically at distinguishing semantic differences, sometimes rating opposites as highly similar. In contrast, large language models (LLMs) show superior ability to identify true semantic distinctions. A key finding is that for code embeddings, simply switching from cosine similarity to Euclidean distance can dramatically improve performance (24-72%), suggesting that the distance metric is often the problem, not the embedding itself. The study provides a diagnostic framework and practical guidance for selecting more reliable similarity assessment tools.
Measuring how similar two pieces of text or code are in meaning, known as semantic similarity, is a crucial task in many software engineering applications. This includes everything from finding relevant code snippets and recommending APIs to automating code reviews and refactoring tools. While large language models (LLMs) are increasingly being used for these assessments, a recent study from the University of Luxembourg raises important questions about whether these methods truly grasp semantic relationships or merely recognize surface-level patterns.
The research, conducted by Serge Lionel NIKIEMA, Albérick Euraste Djiré, Abdoul Aziz BONKOUNGOU, Micheline Bénédicte Moumoula, Jordan Samhi, Abdoul Kader Kaboré, Jacques Klein, and Tegawendé F. Bissyandé, systematically evaluated 18 different similarity measurement approaches. These included traditional word-based methods, embedding techniques (which represent text/code as numerical vectors), LLM-based systems, and structure-aware algorithms specifically for code. To rigorously test these methods, the researchers developed a unique testing framework that applied controlled changes to text and code, allowing them to observe how each method handled various types of semantic relationships.
The findings revealed significant weaknesses in many commonly used metrics. For instance, some embedding-based methods catastrophically failed, incorrectly identifying semantic opposites as similar up to 99.9% of the time. Even certain advanced transformer-based approaches sometimes rated opposite meanings as more similar than truly synonymous ones. The study pinpointed that the poor performance of embedding methods often stemmed from how they calculated distances between these numerical representations, rather than issues with the underlying representations themselves. A surprising discovery was that simply switching from cosine similarity to Euclidean distance for code embeddings dramatically improved results by 24-66%.
In contrast, LLM-based approaches demonstrated a superior ability to distinguish semantic differences. These models produced low similarity scores (0.00-0.29) for genuinely different meanings, a stark improvement over embedding methods that incorrectly assigned high scores (0.82-0.99) to dissimilar content.
The study also delved into the nuances of LLM performance, investigating how prompt design and temperature settings affect their accuracy and consistency. It found that simple prompts were often most effective for detecting semantic differences, while few-shot prompting (providing a few examples) offered the highest consistency. Interestingly, the ‘temperature’ setting, which controls the randomness of an LLM’s output, had minimal impact on semantic evaluation performance, suggesting that model architecture plays a more significant role than this parameter.
Also Read:
- The True Cost of AI in Software Engineering: Introducing SWE-Effi
- Understanding Large Language Models in Legal AI: A Deep Dive into Current Trends and Future Paths
This research provides critical insights for developers and researchers in software engineering. It highlights that current embedding-based similarity metrics frequently misclassify semantic relationships, leading to unreliable tools. However, it also offers actionable guidance: for code similarity applications, evaluating alternative distance metrics like Euclidean distance can lead to substantial improvements with minimal effort. Furthermore, LLM-based approaches offer a more robust understanding of semantics, though careful prompt optimization is key to leveraging their full potential. The full research paper can be accessed here: How Small Transformation Expose the Weakness of Semantic Similarity Measures.
