TLDR: PerQ is a novel, computationally efficient metric for evaluating the personalization quality of multilingual texts generated by language models. It trains a smaller model using majority-voted scores from multiple large language models, significantly reducing evaluation costs and time (up to 96x faster with 12x less GPU memory) compared to direct LLM meta-evaluation. A case study demonstrates its utility in comparing LLM personalization capabilities across different model sizes, generation methods, languages, and target platforms.
In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) are becoming increasingly adept at generating diverse and high-quality text. However, a significant challenge remains: how do we effectively evaluate the quality of these generated texts, especially when it comes to nuanced aspects like personalization? Traditional methods often fall short, and relying solely on powerful LLMs for evaluation can be both expensive and prone to their inherent biases.
Researchers Dominik Macko and Andrew Pulver from the Kempelen Institute of Intelligent Technologies have introduced a novel solution called PerQ, designed for the efficient evaluation of multilingual text personalization quality. Their work addresses the critical need for resource-efficient metrics in an area where human evaluation is time-consuming and general-purpose LLMs are resource-intensive.
The Challenge of Text Personalization Evaluation
Evaluating how well a text is tailored for a specific audience or platform—its personalization quality—is a complex task. Existing metrics like BLEU or ROUGE require reference texts, which aren’t always available for purely generated content. While some newer methods use LLMs as evaluators, this approach can be costly, especially when multiple LLMs are used to mitigate individual biases, as is often recommended for robust meta-evaluation.
Introducing PerQ: An Efficient Solution
PerQ offers a computationally efficient, reference-free method to assess personalization quality. Instead of using large LLMs for every evaluation, PerQ trains a smaller, fine-tuned language model to act as a classifier. This smaller model learns to predict personalization scores based on a dataset that has been carefully annotated. The annotation process itself leverages multiple diverse LLMs, which provide initial scores that are then combined through majority voting. This initial, one-time use of larger LLMs for dataset creation helps to reduce their internal biases, creating a more reliable training ground for PerQ.
Once trained, PerQ can predict personalization quality scores with remarkable speed and significantly lower computational costs. This means researchers can get quick feedback on text quality, accelerating exploratory research and reducing the waste of valuable computing resources.
Building the Foundation: A New Dataset
A key step in developing PerQ was the creation of a new, comprehensive dataset, as no suitable one existed. The researchers generated over 25,000 personalized texts across seven languages (English, German, French, Italian, Slovak, Russian, and Hungarian) for three social media platforms (Twitter/X, Telegram, and Signal). These texts were generated using various large and small LLMs. Subsequently, three different LLMs were employed to meta-evaluate these generated texts against a detailed 4-point scoring schema, with majority voting used to assign a final personalization score to each text.
Performance and Efficiency Gains
The experimental results demonstrate PerQ’s effectiveness. The trained metric achieved an accuracy well above chance, with a strong Spearman correlation coefficient (above 0.8) between its predictions and the majority meta-evaluation scores from the larger LLMs. Crucially, PerQ offers substantial efficiency improvements. Compared to direct meta-evaluation using multiple large LLMs, PerQ can achieve up to a 96-fold speedup in evaluation time while requiring over 12 times less GPU memory. This translates to evaluating personalization quality in minutes rather than hours, significantly contributing to energy sustainability in research.
Insights from the Case Study
The paper also presents a case study using PerQ to answer several research questions about LLM personalization capabilities:
- Small vs. Large LLMs: The study found that smaller LLMs generally produce lower quality personalized texts compared to their larger counterparts.
- Text Generation vs. Modification: Purely generating personalized content tended to result in slightly higher personalization quality than modifying existing texts.
- Language and Platform Differences: English texts showed the best personalization quality, likely due to LLMs’ predominant proficiency in English. Hungarian and Slovak texts exhibited the lowest quality. Among platforms, personalizing for Twitter and Telegram was found to be easier for LLMs than for Signal.
Also Read:
- MENLO: A New Framework for Evaluating and Improving Multilingual LLM Quality
- Unpacking Self-Bias in AI-Generated Translation Benchmarks
Conclusion
PerQ represents a significant advancement in the evaluation of multilingual text personalization quality. By offering a highly efficient and accurate alternative to direct LLM meta-evaluation, it empowers researchers to quickly assess and improve the personalization capabilities of language models, all while conserving valuable computational resources. For more details, you can refer to the full research paper: PerQ: Efficient Evaluation of Multilingual Text Personalization Quality.
