TLDR: A new research paper introduces an AI-driven method using Large Language Models (LLMs) to measure elite political polarization in parliamentary speeches. By analyzing how politicians mention and express sentiment towards opponents, the “Elite Polarization Score” tracks mutual out-party hostility across countries and over time, offering a more nuanced understanding of political divisions. Case studies in Hungary, the UK, and Italy demonstrate its validity in reflecting party-level dynamics and government changes.
A new research project introduces a novel method to quantify elite polarization, focusing on the interactions and emotional evaluations among politicians within parliamentary speeches. This innovative approach harnesses the power of artificial intelligence, specifically Large Language Models (LLMs), to identify instances where politicians refer to one another, ascertain who is speaking and who is being addressed, and gauge the emotional intensity behind these evaluations. The outcome is a comprehensive map of how political elites assess their opposing parties, leading to the creation of an index that measures mutual out-party hostility, effectively defining elite polarization.
For decades, the study of political polarization has been a cornerstone of research on party politics and regime change. However, measuring this phenomenon, particularly among political elites, has presented significant challenges. Traditional methods, such as aggregating survey data, often suffer from limitations in scope, typically being confined to specific countries and infrequent time points. Similarly, older sentiment or contextual analysis techniques demanded extensive training and manual coding, restricting most projects to single-country studies. The recent advancements in artificial intelligence, especially Large Language Models, are now bridging this methodological gap, enabling more expansive and detailed cross-country investigations.
The core of this new methodology is a sophisticated, multi-stage algorithm driven by artificial intelligence. It begins by standardizing parliamentary speeches from existing datasets and submitting them to an LLM via an Application Programming Interface (API). Unlike previous natural language processing (NLP) techniques that relied on predefined keywords, the LLM autonomously identifies every instance where politicians mention their counterparts. It then evaluates the speaker’s sentiment toward the referenced entity, drawing on deep contextual cues rather than simple word matches. A key feature is the LLM’s ability to explain the reasoning behind its interpretations in natural language, enhancing transparency and replicability. Crucially, metadata regarding both the speaker and the addressed entity is meticulously retained.
Following the initial LLM analysis, additional algorithms classify the diverse list of referred political actors into standardized categories, such as political parties, government bodies, or institutions. These algorithms also associate politician names with their respective parties based on the speech date and standardize various spellings or titles that refer to the same entity. Finally, hundreds of thousands of these individual mentions, or ‘dyads’ of MPs referring to each other with various sentiments, are aggregated into a comprehensive time-series index of elite polarization. This index can be aggregated by party and quarter, offering a highly granular and dynamic view of polarization trends.
The adoption of LLMs offers several critical advantages over earlier machine learning techniques and dictionary-based analyses. Comparative studies have shown that pre-trained general AI models achieve higher accuracy than previous generations of neural language models designed for specific tasks. This method eliminates the need for extensive training, making it more accessible to political scientists rather than solely computer science experts. Furthermore, LLMs can operate seamlessly across different languages without additional training, which is vital for creating robust cross-country datasets. Trials with various LLM versions, including models from the ChatGPT-4 family, LLAMA 3.1 70B, and Mistral Small 22B, demonstrate their capability to capture textual nuances previously unmanageable with older techniques. They can distinguish between important references to political figures and mere procedural mentions, and differentiate emotions tied to a specific issue from those directed at an opponent.
The paper defines elite polarization as the collective out-party evaluations within policy debates, essentially analyzing how Members of Parliament (MPs) discuss their opposing counterparts in official speeches. The Elite Polarization Score (EPS) is calculated by aggregating the annual evaluations of out-parties, weighted by the proportion of references each party receives in parliamentary speeches over a given period. This weighting mechanism, based on mention frequency rather than fluctuating vote shares, provides a more stable and fine-grained index that reflects how elites direct their positive or negative evaluations over time.
To assess the validity of this novel concept and methodology, the approach was applied to three European nations: Hungary (2002-2023), the United Kingdom (1987-2020), and Italy (2013-2022). In Hungary, the index revealed a slight increase in Fidesz’s polarization before the 2006 elections, and a surprising decrease during their period of executive aggrandizement (2010-2014), while the opposition MSZP’s scores significantly increased. Overall, elite polarization in Hungary saw a sharp rise after the executive aggrandizement was completed. In the UK, the Labour Party generally exhibited higher polarization than the Conservatives, but notably, Labour’s polarization levels decreased significantly while they were in power (1997-2010). The aggregated UK elite polarization showed a U-shaped trajectory, declining during the New Labour years and rising again with the onset of Brexit. The index also demonstrated a notable correlation with existing measures of ideological polarization. In Italy, party polarization consistently fluctuated in response to parties entering or leaving government coalitions. Parties in opposition, such as the Five Star Movement (M5S) initially, and later the Democratic Party (PD) and Lega, displayed higher levels of polarization, which subsequently decreased when they joined governing coalitions. This responsiveness to government changes further validates the index.
Also Read:
- Rethinking Toxicity: How Diverse Perspectives Improve Online Safety Models
- AI-Powered Survey Validation: Enhancing Psychometric Item Quality with Virtual Respondents
While the index demonstrates strong face validity, reacting to significant events like electoral campaigns and party-level crises, it occasionally does not reflect major country-level events, such as the Brexit crisis or the 2006 Hungarian political crisis. This might be attributed to the often-choreographed nature of parliamentary rhetoric. Nevertheless, this research represents a significant leap forward in political science, providing a robust, cross-national, and time-series measure of elite polarization. It underscores the transformative potential of LLMs in conducting fine-grained political text analysis, paving the way for a new generation of comparative politics datasets and deeper insights into elite interactions. For a comprehensive understanding of the methodology and findings, the full research paper can be accessed here: Elite Polarization in European Parliamentary Speeches: a Novel Measurement Approach Using Large Language Models.
