TLDR: A new research paper explores the use of Large Language Models (LLMs) for structured resource allocation within a Participatory Budgeting framework. The study evaluates LLMs’ ability to select projects under budget constraints using various prompting strategies and, notably, their capacity to infer community preferences from natural language or metadata. The findings suggest that LLMs perform well, sometimes even better, when interpreting unstructured preference data, highlighting their potential as ‘social planners’ in complex decision-making scenarios.
Large Language Models, or LLMs, are becoming increasingly capable of handling complex tasks, including decision-making. However, their ability to manage structured resource allocation, like distributing limited funds or resources, has not been widely explored. A new research paper introduces an innovative approach using Participatory Budgeting (PB) as a framework to test and evaluate how well LLMs can perform in these real-world scenarios.
Participatory Budgeting is a process where community members decide how to allocate a public budget. This study uses PB not only as a practical setting for LLM-based resource allocation but also as a dynamic way to benchmark their reasoning abilities. Unlike traditional static benchmarks, PB tasks are adaptive, requiring LLMs to handle changing factors such as budget constraints, project conflicts, or shifting community preferences.
The researchers tasked LLMs with selecting project subsets under budget limitations, using three main prompting strategies: a straightforward greedy selection, a direct optimization approach, and a refinement method inspired by hill-climbing algorithms. The performance of these LLM-generated allocations was then compared against an ‘oracle’ – a theoretical ideal that maximizes utility.
A particularly interesting aspect of this research is whether LLMs can infer structured preferences from natural language input or community metadata, even without explicit votes. By comparing allocations based on these inferred preferences to those derived from actual, ground-truth votes, the study evaluates the LLMs’ capacity to extract preferences from open-ended, unstructured information. The findings suggest that the way prompts are designed plays a crucial role, and LLMs show significant promise for designing mechanisms that work with diverse, unstructured inputs.
The study explored three types of PB instances to challenge the LLMs: Plain PB Instances (PPI) with standard numerical voter preferences, Vote-Removed PB Instances (VRPI) where LLMs had to infer preferences from voter metadata (like age, sex, and education), and Natural Language Votes PB Instances (NLVPI) where voter preferences were provided as text descriptions. Surprisingly, the LLMs often achieved higher average utility when dealing with natural language preferences or inferred preferences compared to when they were given numerical matrices.
This indicates that LLMs are not only good at processing natural language but might even perform better when interpreting or inferring preferences from textual inputs than from structured numerical data. This capability, sometimes referred to as ‘Theory of Mind’ inference, suggests that LLMs can predict voter preferences based on project categories and demographic information.
The research also demonstrated the adaptability of their PB framework by introducing additional constraints, such as project conflicts, where certain projects cannot be funded together. This increased complexity allowed the researchers to assess how well LLMs can generalize beyond simple numerical feasibility and adapt to more nuanced allocation scenarios.
Also Read:
- New Insights into Multi-Winner Voting Through Data Analysis
- Addressing Overconfidence in AI Judges: New Metrics and Ensemble Approaches
Overall, this work highlights the potential of LLMs to act as effective ‘social planners’ in complex, preference-driven environments, especially where traditional methods for collecting structured preferences might be difficult or costly. For more details, you can read the full paper here.
