TLDR: A new method called Survey and Questionnaire Item Embeddings Differentials (SQuID) allows neural network embeddings from large language models to accurately recover latent dimensions from psychometric surveys. By subtracting a ‘questionnaire embedding’ from individual item embeddings, SQuID successfully generates negative correlations between items, a crucial aspect of value theory that previous embedding approaches struggled with. Validated against human data from the Revised Portrait Value Questionnaire (PVQ-RR), SQuID-treated embeddings, particularly from the Linq-Embed-Mistral model, showed internal consistency and structural congruence comparable to, and sometimes exceeding, human rater judgments. This advancement offers a cost-effective, scalable, and flexible complementary methodology for psychometrics and social science research.
The field of psychometrics, which focuses on measuring human attributes like personality and values, is experiencing a significant shift with the integration of natural language processing (NLP) technologies. Traditional questionnaire-based methods, while robust, often face challenges such as high costs, scalability issues, and potential data quality problems due to respondent fatigue or lack of motivation. This new research introduces a novel approach called “Survey and Questionnaire Item Embeddings Differentials” (SQuID) that leverages neural network embeddings to overcome these limitations.
SQuID is a methodological advancement designed to enable large language model (LLM) embeddings to accurately recover latent dimensions from psychometric survey items. A key challenge in using embeddings for psychometric analysis has been the difficulty in reproducing negative correlations between items, which are crucial for understanding the nuanced relationships between psychological constructs. Previous attempts often required domain-specific fine-tuning or workarounds that were not universally applicable.
The core innovation of SQuID lies in its straightforward yet effective method: it subtracts an average embedding of all questionnaire items (a “questionnaire embedding”) from each individual item embedding. This process helps to filter out common linguistic features that artificially boost similarity between items, thereby allowing the true semantic differences, including negative correlations, to emerge. This approach is highly flexible, working with any model and embeddings of arbitrary dimensions without the need for extensive prompt engineering or fine-tuning.
To validate SQuID, the researchers applied it to the Revised Portrait Value Questionnaire (PVQ-RR), a widely used and extensively validated survey on human value orientations. Schwartz’s theory of basic human values posits a circumplex structure where values that motivationally oppose each other are negatively correlated and placed on opposite sides of a circle. This makes the PVQ-RR an ideal, albeit challenging, test case for an embedding-based approach that aims to reproduce negative correlations.
The study compared SQuID-treated embeddings from several models, including the top-performing Linq-Embed-Mistral, against human rater judgments. The results were compelling. The Linq-Embed-Mistral model, when processed with SQuID, demonstrated internal consistency (measured by Cronbach’s alpha) that was on par with, and in some cases even surpassed, human data across the 19 value dimensions of the PVQ-RR. Furthermore, the dimension-dimension similarities derived from the embeddings showed a substantial positive correlation of 0.74 with human data, explaining 55% of the variance.
Multidimensional scaling (MDS) was used to visualize the relationships between value dimensions. The MDS configurations generated from the SQuID-treated embeddings successfully replicated the theoretically assumed circular structure of human values, mirroring the patterns observed in human data. After a Procrustes matrix transformation to align the embedding and human configurations, the congruence coefficients were found to be 0.88 for the first dimension and 0.82 for the second, indicating a fair to good similarity between the two distinct data sources.
Also Read:
- Unmasking AI Judges: A New Approach to Detecting LLM-Generated Evaluations
- Unlocking New Abilities: How Reinforcement Learning Helps Language Models Compose Skills
This research highlights that semantic embeddings, when processed with SQuID, can effectively replicate complex psychometric structures previously established through extensive human surveys. The methodology offers significant advantages in terms of cost, scalability, and flexibility, while maintaining comparable quality to traditional methods. It provides a complementary tool for psychometrics and social science research, potentially expanding the scope of human behavior and experience represented in measurement tools. Future research could explore its application in investigating cultural biases within LLMs and adapting it for diverse linguistic contexts. For more details, you can read the full paper here.
