TLDR: A new research paper challenges the common assumption that accurately predicting semantic features from word embeddings means the embeddings truly encode that knowledge. It demonstrates that these prediction methods often reflect geometric similarities within vector spaces and can even “predict” random information, suggesting that current “explainability” methods for AI models might be misleading about how much knowledge is genuinely understood.
In the rapidly evolving world of Artificial Intelligence, Large Language Models (LLMs) have become central to natural language processing, showcasing impressive capabilities. However, understanding how these models achieve such performance, beyond simply processing vast amounts of data, remains a significant challenge. A new research paper titled “Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings” by Hanna Herasimchyk, Alhassan Abdelhalim, Sören Laue, and Michaela Regneri from Universität Hamburg delves into this critical area, specifically focusing on how we interpret the knowledge encoded in word embeddings, which are fundamental components of LLMs.
Challenging the Status Quo of AI Explainability
A popular approach to explain the implicit knowledge within word embeddings is called “property inference.” This method involves mapping these embeddings onto collections of human-interpretable semantic features, often from curated datasets known as feature norms. The prevailing assumption has been that if a model can accurately predict these semantic features from word embeddings, then the embeddings must inherently contain that corresponding knowledge. This paper rigorously challenges this assumption.
The researchers demonstrate that prediction accuracy alone is not a reliable indicator of genuine feature-based interpretability. They show that these methods can successfully “predict” even random information, suggesting that the results are often more influenced by algorithmic limitations and the structure of the data itself, rather than a true understanding of semantic representation within the word embeddings. Consequently, simply comparing prediction performance between different datasets might not accurately indicate which dataset’s knowledge is better captured by the embeddings.
The Experiments: Unveiling Misleading Correlations
To validate their claims, the authors applied two commonly used mapping methods, Partial Least Squares Regression (PLSR) and Feed Forward Neural Networks (FFNNs), to map BERT word embeddings to three different feature norms: McRae, Buchanan (both categorical and sparse), and Binder (continuous and dense). They emphasized the importance of proper hyperparameter tuning, noting that previous studies often overfit their models, leading to misleadingly high performance.
Their detailed experiments revealed several surprising findings:
-
Low Upper Bounds for Sparse Data: For sparse feature norms, the maximum possible prediction quality (the “upper bound”) was found to be very low. The models’ actual performance was often close to this low upper bound, making it difficult to discern how much of the result was due to actual information overlap versus the inherent limitations of the method and data structure.
-
Predicting Randomness: The methods could predict random features to some extent, especially when the original data’s sparsity structure was maintained. This means that a model might appear to be learning something meaningful when it’s merely picking up on statistical regularities of random data.
-
Insensitivity to Core Semantic Corruption: Perhaps most strikingly, corrupting essential linguistic knowledge, such as taxonomic relationships (e.g., changing “raven is a bird” to “raven is a fruit”), had very little impact on the prediction results. This suggests that the methods were not truly capturing the semantic meaning of these features.
-
Misleading Scores for Dense Data: For dense norms, even nonsensical, structured values (like the character count difference between a concept and a feature) could yield high correlation scores, making the evaluation metric unsuitable for truly assessing semantic understanding.
What is Actually Being Explained? Geometric Similarity
The paper argues that these mapping methods primarily explain “geometric similarity” rather than specific property knowledge. They found that the methods are effective at capturing how similar concepts are to each other in the vector space of the embeddings, and how this similarity aligns with the similarity of concepts in the feature norm space. However, this is not the same as understanding the individual features that define those similarities.
For instance, if two concepts (like “raven” and “sparrow”) are close in the embedding space and also share many features in the norm, the model can predict this proximity. But it doesn’t necessarily mean the model understands *why* they are both birds or have wings. The sparsity of categorical norms further complicates this, as many features are unique or very rare, making it hard for the model to learn specific property associations.
Also Read:
- Unveiling Word Meaning Storage in AI Language Models
- Mapping Language Relationships Through AI Models
Implications for AI Interpretability
The findings suggest that the intuitive interpretations of property inference methods might be flawed. High prediction accuracy in these contexts does not automatically imply that the AI model has genuinely learned or encoded the human-interpretable knowledge. Instead, the results are heavily influenced by the mathematical properties of the data and the algorithms themselves.
This research highlights a crucial need for more rigorous evaluation of AI explainability methods. It urges the AI community to look beyond simple prediction scores and develop measures that can truly differentiate between correlation and genuine explanation, especially when assessing how deep learning models understand and represent complex semantic information.
