TLDR: A new study using the EuroParlVote dataset reveals significant gender and political biases in large language models (LLMs) when analyzing European Parliament debates. LLMs frequently misclassify female politicians as male and show reduced accuracy for female speakers. Politically, they favor centrist groups and perform better on far-right than far-left ideologies. Proprietary models generally outperform open-weight alternatives in robustness and fairness, while fine-tuning methods like LoRA did not mitigate gender bias. The research highlights the need for more equitable and context-aware AI in political analysis.
Large Language Models (LLMs) are increasingly being applied to political discourse, but a new study highlights persistent gender and political biases within these powerful AI systems. Researchers introduced EuroParlVote, a unique benchmark dataset, to evaluate how LLMs interpret and predict political behavior in the multilingual, multi-party democratic setting of the European Parliament.
Introducing EuroParlVote: A New Lens on Political Bias
The EuroParlVote dataset is a groundbreaking resource that links over 22,000 European Parliament debate speeches to 969 corresponding roll-call vote outcomes. Crucially, it includes rich demographic metadata for each Member of the European Parliament (MEP), such as gender, age, country, and political group. This comprehensive dataset, covering 24 official EU languages, allows for a detailed, demographically-aware analysis of LLM performance in politically sensitive contexts.
Unpacking Gender Bias in LLMs
The study evaluated state-of-the-art LLMs, including proprietary models like GPT-4o, Gemini-2.5-Flash, and Claude-3.5, alongside open-weight alternatives such as LLaMA-3.2 and Mistral-large, on two key tasks: gender classification and vote prediction.
In the gender classification task, where LLMs predicted MEPs’ gender based solely on their speeches, a consistent male-biased pattern emerged. Female MEPs were disproportionately misclassified as male. For instance, LLaMA-3.2 incorrectly labeled over 71% of female speakers as male. When simulating votes, accuracy dropped significantly when all MEPs were hypothetically assigned the gender ‘female’, while assigning ‘male’ yielded the highest accuracy. Proprietary models generally exhibited lower gender misclassification rates and more balanced performance compared to open-weight models.
Qualitative analysis revealed that LLMs often relied on stereotypical language cues, associating assertive or analytical tones with male speakers and social or environmental concerns with female speakers. Political group affiliation also played a role, with far-left MEPs sometimes predicted as female and conservative MEPs as male, regardless of their actual gender. Age also confounded predictions, with older female MEPs often misclassified as male due to formal speech patterns.
Interestingly, fine-tuning techniques like Low-Rank Adaptation (LoRA) did not reduce gender bias in open-weight models; in some cases, it even amplified gender disparities.
Revealing Political Leanings
The research also delved into the political leanings of LLMs by having them simulate votes. The findings indicated a left-centrist bias across all LLMs. Predictive accuracy was highest for centrist or liberal groups, such as the RENEW group, suggesting their moderate stances are easier for models to simulate. While LLMs showed a general left-leaning tendency, a surprising observation was that far-right parties (like Identity and Democracy Group, ID) were simulated more accurately than their far-left counterparts (like The Left group in the EP – Nordic Green Left, GUE/NGL).
Providing explicit political group identity in the prompt improved model performance, especially for ideologically extreme groups, suggesting that explicit context can help mitigate performance disparities across ideological lines.
Qualitative analysis of voting misclassifications showed that LLMs sometimes over-relied on keywords, conflated negative sentiment with opposition, and failed to detect sarcasm or irony in speeches, leading to incorrect predictions.
The Importance of Context and Language
An ablation study confirmed that the semantic content of debate speeches is crucial for accurate vote prediction. When speeches were removed, model accuracy dropped to near-random guessing, highlighting that LLMs do not simply rely on gender or group priors. Furthermore, the multilingual nature of the European Parliament meant that speeches were in their original languages. The study found that translating speeches into English using various methods (GPT-4o, T5, Google Translate API) consistently resulted in lower accuracy compared to using the original-language inputs, underscoring the value of native-language data.
Also Read:
- Unpacking LLM Effectiveness: Fine-Tuning Outperforms In-Context Learning for Misinformation Detection
- AI Models Systematically Amplify Gender Stereotypes, But Fairness Is Within Reach
Towards Fairer AI in Politics
This study, the first to systematically benchmark both gender and political bias in LLMs within a multilingual parliamentary context, underscores the complexity of assessing and mitigating fairness concerns in AI. While proprietary models demonstrated greater robustness and fairness, open-weight models often struggled in predictable ways. The researchers have released the EuroParlVote dataset and code to support future research on fairness, transparency, and accountability in NLP within political contexts. You can find more details about this research paper here.
