TLDR: A research paper by Maxim Chupilkin reveals that Large Language Models (LLMs) evaluating economic policies prioritize low unemployment, reduced inequality, environmental protection, and financial stability over traditional macroeconomic concerns like economic growth, inflation, and government debt. This ‘left-leaning’ bias is consistent across various LLM models and policy scenarios, highlighting the need for users in economics to be aware of these inherent assumptions when utilizing AI for analysis and recommendations.
As artificial intelligence, particularly large language models (LLMs), becomes increasingly integrated into various fields, including economics, a crucial question arises: How do these AI models approach and evaluate economic policies? A recent research paper titled “Left Leaning Models: AI Assumptions on Economic Policy” by Maxim Chupilkin delves into this very question, aiming to uncover the inherent assumptions and biases within LLMs regarding economic decision-making.
The proliferation of LLMs in economics is undeniable, with applications ranging from summarizing financial texts to offering policy recommendations. However, the internal workings of these models often remain a ‘black box,’ making it difficult to understand the underlying principles guiding their economic assessments. This research addresses that gap by systematically investigating what factors most influence an LLM’s evaluation of economic policies.
To shed light on these assumptions, the paper employs a conjoint experiment, a method popular in social sciences for understanding multi-factor decision-making. The experiment involved presenting LLMs with five different economic policy scenarios: fiscal stimulus, trade liberalization, monetary policy, changes in taxation, and changes in regulation. For each scenario, the predicted outcomes for seven key economic variables were systematically varied: economic growth, income inequality, environmental harm, public debt ratio, inflation, unemployment, and financial stability. These outcomes were presented as either ‘higher’ or ‘lower’ to simplify interpretation.
In total, the LLMs were evaluated across 640 distinct scenarios, with each scenario run 100 times, accumulating a massive dataset of 64,000 observations. The primary analysis was conducted using the OpenAI GPT-4o mini model, chosen for its accessibility and reproducibility. The models were prompted to respond with a score between 0 and 100, indicating their recommendation for adopting a given policy.
Key Findings: What Matters Most to AI?
The research yielded striking and consistent results. Across all scenarios, LLMs were found to be most sensitive to outcomes related to unemployment, income inequality, environmental harm, and financial stability. Changes in these factors had the most significant impact on the models’ policy evaluation scores. For instance, high unemployment drastically reduced a policy’s approval score.
Conversely, traditional macroeconomic concerns such as economic growth, inflation, and government debt were found to be of secondary importance to the LLMs. Surprisingly, economic growth was often the least influential factor in most scenarios. This suggests a prioritization that differs from conventional economic thinking, which often places significant emphasis on GDP growth.
While there was a general consistency, the study also noted that LLMs showed some sensitivity to the specific policy domain. For example, inflation carried more weight in monetary policy scenarios, and public debt was more important in taxation scenarios. This indicates that the models do incorporate some economic logic relevant to the policy context.
Consistency Across Different Models
To ensure the findings were not unique to the OpenAI GPT-4o mini, the fiscal stimulus scenario was replicated across several other prominent LLMs, including OpenAI GPT-4o, Anthropic Claude Haiku 3.5, Anthropic Claude Sonnet 3.5, and Google Gemini 2.0 flash. The results were remarkably consistent across these diverse models from different providers. All tested models prioritized unemployment, followed by inequality, environmental harm, financial stability, and government debt, with inflation and growth being lesser concerns.
This cross-model consistency suggests that the observed biases might be deeply embedded, potentially stemming from similar training data or fundamental architectural instructions, rather than being mere idiosyncrasies of a single model.
Also Read:
- Decoding Political Texts: Advancing AI for Bias Detection
- Unlocking LLM Potential: A New Approach to Positional Bias
Implications: AI’s ‘Left-Leaning’ Economic Stance
The paper concludes that LLMs exhibit a strong preference for policies that lead to low unemployment, reduced environmental harm, increased financial stability, and decreased inequality. This prioritization, particularly the de-emphasis on economic growth and inflation compared to social and environmental factors, suggests that LLMs lean towards what some might describe as a ‘left of center’ orientation in their economic policy evaluations. For more details, you can refer to the full research paper at https://arxiv.org/pdf/2507.15771.
These findings carry significant implications for economists, policymakers, and market participants who increasingly rely on LLMs for analysis and recommendations. It highlights the critical need to understand and account for these inherent assumptions and biases when using off-the-shelf AI models. The research also serves as a proof-of-concept, demonstrating that social science methodologies can be effectively used to open the ‘black box’ of AI and study its decision-making processes, paving the way for future, more complex investigations into AI behavior.
