TLDR: A new research paper evaluates the policymaking capabilities of Large Language Models (LLMs) in addressing homelessness. It introduces a benchmark of 170 decision scenarios across four global cities, grounded in the Capability Approach for human development. The study compares LLM policy choices with human experts and uses an agent-based model to simulate social impacts. Findings show LLMs can offer valuable, scalable policy insights, often prioritizing immediate safety, and can even demonstrate superior overall balance in fulfilling needs in simulations, but require careful contextual calibration and expert collaboration.
Large Language Models (LLMs) are increasingly finding their way into critical sectors, and their potential to assist in complex social policymaking, such as addressing homelessness, is a subject of intense research. A recent study titled “What Would an LLM Do? Evaluating Policymaking Capabilities of Large Language Models” by Pierre Le Coz, Jia An Liu, Debarun Bhattacharjya, Georgina Curto, and Serge Stinckwich delves into this very question, assessing how well LLMs align with human experts and among themselves when it comes to crafting policies for homelessness alleviation.
Homelessness is a global crisis, affecting over 150 million people worldwide. Crafting effective policies to tackle this issue is incredibly complex, requiring an understanding of diverse contextual factors, social dynamics, and the ability to anticipate outcomes in dynamic systems. The researchers highlight that traditional policymaking often struggles with the multifaceted nature of homelessness, which extends beyond material needs to include dignity, social integration, and combating stigmatization.
A Novel Approach to Evaluation
To evaluate LLMs in this sensitive domain, the researchers developed a unique benchmark. This benchmark consists of 170 decision scenarios related to homelessness, spanning four distinct geographies: South Bend (USA), Barcelona (Spain), Johannesburg (South Africa), and Macau SAR (China), plus a universal context. Each scenario presents a dilemma faced by a non-profit organization and offers four policy choices. These policies are not arbitrary; they are grounded in the Capability Approach for human development, a framework that emphasizes opportunities for individuals to lead a meaningful life with dignity, rather than just focusing on material resources.
The study involved prompting various LLMs, including GPT-4.1, Llama 3, and Mixtral, to act as policymakers. They were tasked with selecting a top policy choice and ranking all options, along with providing justifications. This allowed for a direct comparison of LLM decisions with those of human domain experts from Barcelona, Johannesburg, and South Bend.
Bridging Policy and Impact with Simulations
One of the most innovative aspects of this research is the automated pipeline that connects LLM-generated policy proposals to an agent-based model (ABM). This integration allows for the exploration of the social impact of recommended policies through simulated social scenarios. Essentially, the LLMs translate natural language policy proposals into behavioral adjustments for agents within the simulation, which then helps gauge how these policies might affect the needs satisfaction of people experiencing homelessness (PEH).
Key Findings: LLMs vs. Experts
The empirical investigation revealed several interesting insights. Firstly, different LLMs can disagree on their top policy choices and justifications, indicating that the choice of LLM matters significantly in policy recommendations. Secondly, when comparing LLMs with human experts, a moderate but nuanced alignment was observed. While LLMs like GPT-4.1 showed a notable agreement with some experts, sometimes even exceeding human-human agreement baselines, a systematic difference in prioritization emerged.
LLMs tended to prioritize policies that address immediate physical safety, such as life, bodily health, and bodily integrity. For example, in a scenario concerning an open-air drug scene, GPT-4.1 favored a supervised consumption site (health-based harm reduction), whereas a human expert prioritized a policing surge (enforcement-based order restoration). Experts, on the other hand, often balanced immediate safety with long-term empowerment and social cohesion, focusing on capabilities like practical reason (skills training) and affiliation (community integration).
Interestingly, the study found that explicitly prompting LLMs to consider local geographical context led to a closer alignment with local experts’ choices, suggesting that contextual calibration is crucial for LLM deployment in such sensitive areas.
Simulated Social Impact
The ABM simulations, specifically for Barcelona, provided a concrete look at the impact of policies. The results indicated that LLM-recommended policies demonstrated a superior overall balance in fulfilling the needs of PEH agents compared to expert-recommended policies. While both types of policies improved safety and self-esteem, LLM-implemented policies showed more substantial improvements and consistently prevented negative effects on belonging needs that were sometimes observed with expert policies. Both also reduced outcome variability, but LLMs achieved this while maintaining positive or neutral effects across all need categories.
Also Read:
- AI-Powered Scenario Generation for Complex Service Environments
- Auditing AI for Disability Inclusivity: Where Large Language Models Fall Short
The Future of AI in Policymaking
This research highlights the promising potential of LLMs to offer scalable and non-invasive support for social policymaking. By providing alternative, contextualized policies and insights into their potential social impact through simulations, LLMs can be valuable tools for human policymakers. However, the authors emphasize the necessity of responsible guardrails and contextual calibrations, developed in collaboration with local domain experts, to ensure ethical and effective deployment. The study also acknowledges limitations, such as being conducted only in English and having a single expert annotation per location, pointing to areas for future work.
For more details, you can read the full research paper here.
