TLDR: JE-IRT is a novel geometric framework for evaluating Large Language Models (LLMs) by embedding both models and questions in a shared space. It reveals that LLM abilities are multidimensional, with question semantics encoded in embedding direction and difficulty in embedding norm. This framework accurately predicts LLM behavior, scales efficiently to new models, and uncovers an LLM-centric knowledge taxonomy, offering a more nuanced understanding of model capabilities than traditional single-score evaluations.
Evaluating the true capabilities of Large Language Models (LLMs) has long been a challenge. Current methods often condense diverse abilities into single scores, which can obscure the complex, multidimensional nature of how these models understand and respond to questions. A new research paper introduces a novel approach called Joint Embedding Item Response Theory, or JE-IRT, offering a fresh, geometric perspective on LLM evaluation.
The paper, titled “A GEOMETRIC LENS ON LLM ABILITIES THROUGH JOINT EMBEDDING ITEM RESPONSE THEORY,” was authored by Louie Hong Yao, Nicholas Jarvis, Tiffany Zhan, Saptarshi Ghosh, Linfeng Liu, and Tianyu Jiang. It proposes a framework that embeds both LLMs and questions into a shared, high-dimensional space. In this space, the direction of a question’s embedding encodes its semantic meaning, while its norm (or magnitude) reflects its difficulty. The correctness of an LLM’s answer to a question is then determined by the geometric interaction between the model’s and the question’s embeddings.
This geometric approach moves beyond simply ranking LLMs globally. Instead, it allows for the identification of topical specialization, meaning different LLMs might excel in different areas rather than one being universally superior. It also enables smooth variations in predicted responses for semantically related questions, a crucial aspect that traditional evaluation methods often miss.
The researchers conducted experiments that revealed several key insights. They found that out-of-distribution behavior – how an LLM performs on questions outside its typical training data – can be explained by the directional alignment between the model and question embeddings. Furthermore, questions with larger embedding norms consistently proved to be harder for LLMs, validating the framework’s intuitive design.
One of JE-IRT’s significant advantages is its scalability. Once the shared embedding space is learned, integrating new LLMs into the framework is remarkably efficient. It only requires fitting a single embedding for the new model, which can be done with a small fraction of data while achieving performance comparable to full joint training. This is particularly valuable given the rapid pace at which new LLMs are developed and released.
The framework also sheds light on the internal organization of knowledge within LLMs. By clustering the learned question embeddings, the researchers discovered an LLM-internal taxonomy that only partially aligns with human-defined subject categories. This suggests that LLMs categorize and relate information in ways that may differ from our conventional academic structures, offering a deeper understanding of their representational sense.
In contrast to traditional Item Response Theory (IRT), which assigns content-agnostic parameters to questions based solely on response patterns, JE-IRT directly links difficulty and discrimination to the question’s content and semantics. Traditional IRT often assumes a global ordering of LLM abilities, an assumption that the paper demonstrates is frequently violated in practice, even within fine-grained benchmarks.
The implementation of JE-IRT is publicly available, allowing other researchers and developers to explore and utilize this innovative evaluation framework. You can find more details in the full research paper: A GEOMETRIC LENS ON LLM ABILITIES THROUGH JOINT EMBEDDING ITEM RESPONSE THEORY.
Also Read:
- Evaluating Language Model Text Quality Through Internal Geometric Properties
- RADAR: Intelligent Routing for Reasoning LLMs Balances Performance and Cost
This work establishes an interpretable geometric lens that connects LLM abilities with the inherent structure of questions, providing a distinctive and powerful perspective on model evaluation and generalization. It opens new avenues for understanding how LLMs truly operate and how their diverse capabilities can be more accurately assessed and leveraged.
