TLDR: A study investigated how prompt language and explicit cultural framing influence the cultural values expressed by 10 Large Language Models (LLMs) across 11 languages using established value surveys. It found that while prompting can induce variation, it largely fails to overcome a systematic bias towards the values of a few Western and prosperous countries (Netherlands, Germany, US, Japan). LLMs tend to give neutral or progressive responses and show more similarities than differences across models, highlighting that current multilingual LLMs are not yet truly multicultural.
Large Language Models (LLMs) are becoming ubiquitous, adopted by users across the globe who interact with them in a multitude of languages. However, a critical question arises: can these powerful AI systems truly represent the diverse cultural values of their vast user base? A recent study delves into this complex issue, examining how the language used in prompts and explicit cultural framing influence LLM responses and their alignment with human values in different countries.
The research, titled LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framing, was conducted by Bram Bulté and Ayla Rigouts Terryn. Their comprehensive investigation involved probing 10 different LLMs with 63 items from two well-established surveys: the Hofstede Values Survey Module and the World Values Survey. These survey items were translated into 11 languages and presented to the LLMs with and without explicit cultural perspectives.
The study confirmed that both the language of the prompt and the explicit cultural perspective provided do indeed produce variations in the LLMs’ outputs. This suggests that LLMs are responsive to these input changes. However, this responsiveness comes with a significant caveat: while targeted prompting can, to some extent, guide LLM responses towards the predominant values of corresponding countries, it does not overcome a systematic bias. The models consistently leaned towards values associated with a restricted set of countries in the dataset: the Netherlands, Germany, the United States, and Japan.
Interestingly, all tested models, regardless of their origin, exhibited remarkably similar patterns. They tended to produce fairly neutral responses on most topics, but showed selective progressive stances on issues like social tolerance. The researchers found that explicitly framing a cultural perspective improved alignment with human cultural values more effectively than simply using a targeted prompt language. Unexpectedly, combining both approaches (using a targeted language with an explicit cultural frame) was no more effective than using cultural framing with an English prompt.
These findings highlight an uncomfortable middle ground for LLMs: they are flexible enough to show variation based on prompt changes, yet too firmly anchored to specific cultural defaults to adequately represent the full spectrum of cultural diversity. The study’s authors suggest that this bias points towards a tendency for LLMs to align with ‘secular-rational’ and ‘self-expression’ values, which are characteristic of Western, secular, and more prosperous societies.
The research also touched upon the presence of cultural stereotypes in LLM outputs. Instances of extremely stereotypical descriptions were observed when models were asked to respond from specific cultural perspectives. For example, a Dutch persona might mention cycling past cheese factories and windmills, while a Russian persona might begin responses with ‘Comrade’. This indicates that LLMs might be reproducing external, often simplified, views of cultures rather than deeply understanding their nuanced values.
A crucial implication of this study is a caution against using LLMs as ‘synthetic social agents’ to simulate human survey responses. The systematic cultural bias observed in LLMs means they may misrepresent and ‘flatten’ identity groups, inherently favoring majority representations present in their training data. This raises significant ethical concerns about the use of AI in social science research.
Also Read:
- New Benchmark Reveals LLMs Struggle to Grasp Deep Human Values, Favoring Surface Preferences
- Bridging the Language Divide: How Low-Resource Languages Impact Global AI Adoption
In conclusion, while LLMs possess impressive multilingual capabilities, they are not yet truly multicultural. The study underscores the need for continued research into how LLMs encode and represent cultural values, especially as their global adoption continues to grow. Addressing these challenges is vital for ensuring that AI technologies can fairly and accurately serve a diverse global user base.
