TLDR: This paper introduces QASU, a new benchmark to evaluate how well Large Language Models (LLMs) understand and process questionnaire data. It tests six structural skills across different data formats and prompting strategies. The study found that the way questionnaire data is formatted and how prompts are designed significantly impacts LLM accuracy, with improvements of up to 8.8% possible. It also shows that self-augmented prompting can further boost performance by 3-4% on average, providing practical guidance for integrating LLMs into survey analysis.
Surveys are a cornerstone of modern data collection, from market research and academic studies to healthcare feedback and customer satisfaction forms. Millions of people complete them daily, generating vast datasets rich with insights. However, the unique structure and sheer scale of this questionnaire data pose a significant challenge for large language models (LLMs), which are otherwise adept at understanding and reasoning over open-ended text.
Traditional survey analysis tools like Qualtrics, SPSS, and REDCap are primarily designed for human interaction, creating a barrier to seamless integration with LLMs and AI-powered automation. This gap leaves researchers and users without clear guidance on how to best prepare questionnaire data for LLM consumption.
Introducing QASU: A New Benchmark for LLM Structural Understanding
To address this, researchers have introduced QASU (Questionnaire Analysis and Structural Understanding), a groundbreaking benchmark designed to evaluate LLMs’ ability to process and understand questionnaire data. QASU focuses on six key structural skills, including looking up answers, counting respondents, and performing multi-step inferences. It tests these skills across six different data serialization formats and multiple prompting strategies.
The study’s experiments with contemporary LLMs, including GPT-5-mini and Gemini-2.5-Flash, revealed crucial insights. One of the most significant findings is that the choice of data format and prompt combination can dramatically impact accuracy, leading to improvements of up to 8.8% compared to less optimal approaches. For instance, HTML format often outperformed Turtle (TTL) in answer lookup tasks. Furthermore, simply removing partition marks (visual separators in the data) could degrade performance by 16-24% across various tasks, highlighting the importance of clear structural cues.
The Power of Prompting: One-Shot and Self-Augmented Approaches
The research also emphasized the critical role of prompting strategies. One-shot prompting, where the LLM is given a single example before being asked to perform a task, consistently outperformed zero-shot approaches (no examples). This gap was particularly noticeable in tasks requiring multi-step reasoning, with up to a 25% degradation in accuracy for zero-shot methods.
A novel aspect explored was self-augmented prompting, where the LLM first generates its own auxiliary notes or interpretations about the data before providing a final answer. This lightweight structural hint yielded further improvements of 3-4% on average, especially benefiting tasks like reverse look-ups and conceptual aggregation. This suggests that allowing the model to ‘pre-process’ the input in its own words can help it adapt better to the nuances of specific datasets.
Key Structural Skills Evaluated by QASU
QASU evaluates LLMs on fundamental operations crucial for any questionnaire analysis:
- Answer Lookup: Retrieving a specific answer for a given respondent and question.
- Reverse Answer Lookup: Finding all respondents who provided a specific answer to a question.
- Respondent Count: Determining the total number of respondents or those meeting a global criterion.
- Conceptual Aggregation: Counting respondents whose answers fall within a defined high-level concept.
- Multi-hop Relational Inference: Identifying respondents who meet multiple conditions across different questions.
- Rule-based Querying: Finding respondents based on numerical predicates (e.g., greater than, less than).
These tasks progress from simple retrieval to more complex reasoning, allowing researchers to understand how different input designs affect an LLM’s ability to align natural language queries with data, resolve paths, and combine multiple constraints reliably.
Also Read:
- Unpacking AI’s Ability to Translate Tables into Natural Language
- Efficient LLM Evaluation: A New Item-Centric Approach with Cognitive Scales
Practical Implications for LLM-based Questionnaire Analysis
The findings from QASU offer concrete, evidence-based guidance for integrating LLMs into real-world questionnaire analysis. By identifying effective serialization and prompting strategies, the benchmark provides a clear pathway for researchers and practitioners in fields like medical research, social sciences, and market analysis to confidently leverage LLMs for tasks such as generating executive summaries, identifying recurring complaints, or inferring consumer trends from survey data.
This work bridges the gap between academic standards and the practical realities of diverse questionnaire types, offering a simple yet versatile foundation for advancing both research and real-world practice in LLM-based questionnaire analysis. For more detailed information, you can read the full research paper here.
