TLDR: A research paper explores how Large Language Models (LLMs) like GPT-4 Turbo can predict which public projects will be funded in Participatory Budgeting (PB) initiatives. The key innovation is a privacy-preserving approach that uses only project descriptions and anonymous historical voting data, avoiding sensitive voter demographics. The study found that LLMs, particularly when given historical context, outperform traditional machine learning methods in predicting project rankings and top-funded projects, offering a low-barrier tool to improve transparency and civic engagement in PB.
Participatory Budgeting (PB) is a democratic process where citizens actively propose and vote on public investment projects. While it holds great potential for empowering communities, PB initiatives often face challenges such as low participation rates and the difficulty of managing a large volume of project submissions. These issues can limit the visibility and perceived legitimacy of the process.
A recent research paper introduces an innovative approach to strengthen PB elections. The study focuses on two key areas: assisting project proposers in creating more effective proposals and helping PB organizers efficiently manage numerous submissions with transparency. The core of their method is a privacy-preserving system designed to predict which PB proposals are likely to receive funding. Crucially, this system relies solely on the textual descriptions of projects and anonymous historical voting records. It deliberately avoids using voter demographics or any personally identifiable information, addressing a significant concern in digital democracy.
The researchers evaluated the performance of GPT-4 Turbo, a state-of-the-art Large Language Model (LLM), in forecasting proposal outcomes across different scenarios. Their findings indicate that while the LLM’s existing knowledge is helpful, it needs to be supplemented with past voting data to generate predictions that accurately reflect real-world PB voting behavior. This highlights the importance of historical context for the AI to make relevant predictions.
The study’s contributions are significant. They constructed and released two new public datasets from multi-year PB processes in Toulouse, France, and Wrocław, Poland, including consolidated voting data, project titles, and descriptions in both original languages and English. They demonstrated that GPT-4 Turbo can effectively predict PB election outcomes using only public data. Furthermore, their LLM-based pipeline was shown to outperform traditional machine learning models like K-Nearest Neighbors (KNN) and probabilistic models. Remarkably, it achieved performance comparable to other LLM-based approaches that *do* use voter demographics, proving that accurate predictions are possible without compromising privacy.
The methodology involved rigorous data cleaning and preprocessing. To ensure the LLMs had no prior exposure to the specific voting results or full project descriptions, the researchers conducted tests, confirming the models could not reconstruct original project details or costs from titles alone. This validation step is vital for the integrity of their prediction tasks.
The researchers assessed their predictions using three metrics: the number of votes, the ranking of projects (from most to least voted), and the accuracy of identifying the top-k funded projects. They compared LLM performance across three prompt variants: ‘No-context’ (without past election results), ‘Retrieval-Augmented Generation’ (RAG, with a summary of past results), and ‘In-context’ (with full past project data and votes). They also incorporated Chain-of-Thought and Step-Back reasoning techniques to enhance LLM performance.
The results showed that while traditional machine learning models sometimes performed better at predicting the exact number of votes, LLM-based methods, particularly RAG and In-context variants, significantly outperformed them in predicting the ordinal ranking and identifying the top-k projects. This is crucial because many PB selection mechanisms, like greedy algorithms, primarily rely on project rankings. The ‘No-context’ LLM variant performed poorly, underscoring that general prior knowledge about a city is insufficient without specific historical voting data.
This research underscores the potential of AI-driven tools to support PB processes by enhancing transparency, improving planning efficiency, and boosting civic engagement. The approach has low data-collection barriers, making it practical for municipalities. The paper also notes that LLMs are robust to language differences, as experiments with translated datasets yielded similar results. However, the authors acknowledge limitations, such as the commercial nature of GPT-4 Turbo and the need to explore strategic user behavior and potential LLM biases. They emphasize that their findings are an exploratory proof-of-concept, requiring further scrutiny before real-world deployment.
Also Read:
- AI Models Master Community Resource Allocation Through Participatory Budgeting
- Assessing AI’s Readiness for Critical Social Decisions: A Look at Homelessness Resource Allocation
For more details, you can read the full research paper: Leveraging LLMs for Privacy-Aware Predictions in Participatory Budgeting.
