TLDR: A study found that Large Language Models (LLMs) overuse certain words like “delve” and “nuanced” because human evaluators, during Learning from Human Feedback (LHF) training, subtly prefer texts containing these words. This preference, scaled up, leads to the AI’s distinct lexical style, suggesting LHF is a key cause of this “lexical bias” and highlighting a potential misalignment between LHF workers’ preferences and general user expectations.
Large Language Models (LLMs) have become ubiquitous, but many users have noticed a peculiar habit: these advanced AI systems tend to overuse certain words. Terms like “delve,” “intricate,” and “nuanced” frequently appear in AI-generated text, leading to a distinct, sometimes repetitive, style. For a long time, the exact reasons behind these lexical choices remained unclear. A recent research paper, “Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback,” sheds light on this phenomenon, pointing to Learning from Human Feedback (LHF) as a significant contributor. You can find the full paper at https://arxiv.org/pdf/2508.01930.
Understanding Learning from Human Feedback (LHF)
LHF is a crucial post-training procedure for LLMs, designed to align their outputs more closely with human preferences. It encompasses methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). During LHF, human evaluators provide feedback, often through A/B testing or ranking different text variants, guiding the model to produce more desirable outputs. While LHF has been instrumental in the success of models like ChatGPT, the lack of transparency surrounding the specific procedures and datasets used makes it challenging to research its precise effects.
Investigating Lexical Overuse: A Two-Step Approach
The study, conducted by Tom S. Juzek and Zina B. Ward from Florida State University, tackled the potential link between LHF and LLM lexical choices through a clever two-step process.
First, they developed a straightforward method to identify words potentially favored by LHF. They compared the language outputs of two versions of Meta’s Llama model: Llama Base (trained without LHF) and Llama Instruct (trained with LHF, specifically Direct Preference Optimization). By analyzing millions of words generated by both models based on PubMed abstracts, they found significant differences in word usage. Words like “nuanced” (adjective, +8342% increase), “nuance” (verb, +6301%), “firstly” (adverb, +4794%), “reliance” (noun, +3193%), and “generalizability” (noun, +3124%) were dramatically more frequent in the Llama Instruct model’s output. Importantly, the researchers noted a considerable overlap between these LHF-promoted words and those commonly identified in academic literature as being overused by LLMs compared to human-generated text.
Second, to provide more conclusive evidence, the researchers conducted an experiment that emulated the LHF procedure. They created pairs of text variants, where one variant contained more of the LHF-identified words and the other contained fewer, while keeping other factors like length and content as similar as possible. They then recruited 400 participants, primarily from countries in the Global South where English is widely used, to rate their preference between these text pairs. This recruitment strategy aimed to reflect the demographics of typical LHF workers.
Key Findings: Human Preference for “AI-Like” Language
The results were striking: participants showed a statistically significant preference for text variants that contained more of the words identified as potentially promoted by LHF. Specifically, 52.4% of choices favored the high LHF-Score variants over the low LHF-Score variants (47.6%). This finding strongly suggests that human evaluators, when providing feedback, subtly prefer texts that include these specific lexical items, and these preferences, when scaled up through LHF training, become entrenched in the LLM’s output, leading to the observed lexical overuse.
An interesting side observation was made regarding the adjective “nuanced.” While overall high LHF-Score variants were preferred, items containing “nuanced” saw a markedly lower preference (46.6%) compared to those without it (54.5%). This hints that some words, despite being LHF-promoted, might eventually lead to user dispreference if overused too much, similar to the backlash against “delve.”
Also Read:
- The AI Echo: How Large Language Models Are Shaping Spoken English
- AI Models Exhibit Human-Like Risk Biases, Influenced by Language Context
Implications and Future Directions
This research provides compelling evidence that LHF is a primary driver behind the distinctive lexical patterns seen in LLMs. The study highlights a potential “misalignment” between the lexical expectations of LHF workers and general LLM users. The authors suggest that the preferences of LHF workers, who are often young and located in the Global South, might be influencing the language of LLMs. This could be seen as an accelerated form of intergenerational language change, or it might be a result of the LHF task itself, where evaluators might prioritize certain words as proxies for quality when quickly skimming text.
The study’s findings have important implications for LLM development. If developers wish to mitigate this lexical overuse, interventions could be targeted at LHF datasets. This might involve diversifying the human evaluator workforce or adjusting datasets post-collection to ensure a greater balance of lexical preferences. The paper also underscores the broader importance of transparency in LLM development, particularly regarding LHF procedures and datasets, to better understand and align these powerful language technologies with diverse human expectations.
