TLDR: A study analyzed Large Language Model biases using controversial Romanian historical questions across Romanian, English, Hungarian, and Russian. It found that LLMs exhibit significant representational instability, reflecting cultural and geopolitical biases from their training data, and lack consistent historical understanding, making them unreliable for sensitive historical debates. The research highlights the need to prioritize consistency and bias in LLM evaluation.
The research paper, “A Cross-Lingual Analysis of Bias in Large Language Models Using Romanian History,” by Matei-Iulian Cocu, Răzvan Cosmin Cristia, and Adrian Marius Dumitran, delves into the fascinating and critical issue of bias in Large Language Models (LLMs). This study uses controversial Romanian historical questions to explore how LLMs respond across different languages and contexts, ultimately aiming to understand their inherent biases.
The motivation behind this research stems from the understanding that history is often presented through varied perspectives, heavily influenced by a state’s culture and ideals. Since LLMs are trained on vast datasets that might contain ambiguities or reflect specific viewpoints, they can inadvertently instill a lack of neutrality in their users. The authors highlight that reasoning, a prominent feature of LLMs, often comes with a bias towards certain ideologies, especially in fields like history, which have been frequently reinterpreted.
Methodology: A Three-Phase Approach
The methodology for this study was meticulously structured into three key phases. The first phase established a linguistic framework using four languages: Romanian (as the native baseline), English (for its global, often Western-centric perspective), Hungarian (due to significant political and historical tensions with Romania), and Russian (to examine the influence of a major regional power). This diverse selection aimed to probe for bias from distinct cultural and historical angles.
In the second phase, 14 statements concerning debated historical events and periods were selected. These included topics like the dispute over Transylvanian land and the Fall of Communism. To ensure historical validity and neutrality, these statements were developed in consultation with a professional medievalist, ensuring they represented genuine points of debate rather than open-ended questions.
The third phase involved a three-layered questioning process to deconstruct model bias and response inconsistency. Initially, models were forced to give a simple “Yes” or “No” answer, establishing an absolute stance. Next, they were asked to provide a numerical value on a 1-10 Likert-type scale, measuring their conviction and identifying “stance reversals.” Finally, for the most intricate stage, LLMs had to elaborate a full-scale structured essay. To evaluate these complex outputs, a powerful LLM was assigned the role of “LLM-as-a-judge,” rating the nuance, neutrality, and factual accuracy of each response on a 1-10 scale. This multi-stage approach allowed for a comprehensive analysis of how prompt format and language influence model reasoning and adopted beliefs.
The study selected 13 diverse LLMs, varying in architecture, parameter scale, and developer origin, including models from the Deepseek and Llama families. Queries were run using consistent prompt templates for binary, numerical, and essay-type answers, as well as for the LLM-as-a-judge evaluation.
Key Findings: Instability and Cultural Artifacts
The results revealed several critical insights. Consistency within models varied significantly; while some smaller LLMs with superior fine-tuning capabilities showed high consistency (e.g., LLama-4-Scout-17B-16E-Instruct, gemma-3n-E4B-it), others like Deepseek’s suite and OpenAI’s gpt-oss-20b struggled, particularly with Romanian queries, indicating linguistic deficits.
Regarding consistency within languages, the study found distinct patterns of linguistic divergence on contentious topics. For historically consensual affirmations (e.g., Ottoman rule, Holocaust), all languages showed near-perfect agreement. However, for questions tied to national narratives, significant outliers emerged. Russian, for instance, showed extremely low agreement on statements about Ceaușescu and Daco-Romanian continuity, reflecting a strong, divergent perspective in its training datasets. Similarly, Hungarian deviated significantly on Transylvanian demography, indicating a counter-narrative. This highlights that LLMs reflect the biases and dominant narratives present in their language-specific training data.
Cross-language analysis further showed that Hungarian had the largest divergences with Romanian on historiographical matters. A question about the “hu kingdom” (referring to Moldavia and Wallachia breaking away from Hungary) also stood out, with most models ignoring details in the Romanian linguistic context. The study also explored the “meticulous effect of temperature,” confirming that lower temperatures generally lead to better results, and models with existing consistency problems show even steeper shifts with temperature changes.
Also Read:
- How Language Models Learn to Balance Internal Knowledge with New Information
- Conversational AI Robustness: How Semantic Shifts Impact LLM Reliability Over Time
Conclusions: LLMs as Malleable Narrative Engines
In conclusion, the research demonstrates that LLMs are not stable repositories of historical facts but rather malleable narrative engines highly sensitive to query format and linguistic context. The findings point to three primary conclusions:
1. Representational instability: A model’s stance is not fixed but contingent on prompt structure, with frequent “stance reversals” between binary and scaled formats.
2. Cultural artifacts: LLMs encode and reproduce dominant historiographical and geopolitical biases of their language-specific training data, as seen in divergences between Romanian, Hungarian, and Russian answers.
3. Lack of epistemic certainty: High variability across identical runs, even for top-tier models, indicates that LLMs “calculate the most probable continuation of a sequence” rather than truly “knowing” history. This stochasticity makes them unreliable for sensitive historical debates.
The authors argue that future LLM evaluation must prioritize consistency and bias measurement as first-class criteria for trustworthiness. Future work includes investigating capabilities across different model scales and architectures, further exploring the “LLM-as-a-judge” paradigm to create high-quality datasets, and fine-tuning models on foundational and advanced historical materials to instill multi-perspective understanding.
For more details, you can read the full paper here: A Cross-Lingual Analysis of Bias in Large Language Models Using Romanian History.
