TLDR: POLIS-Bench is a novel evaluation suite for Large Language Models (LLMs) in bilingual (Chinese/English) governmental policy scenarios. It introduces an up-to-date policy corpus, three scenario-based tasks (Clause Retrieval & Interpretation, Solution Generation, Compliance Judgment), and a dual-metric evaluation combining semantic similarity and LLM.Judge accuracy. The research found that reasoning models generally outperform chat models, compliance tasks are the most difficult, and fine-tuning with POLIS-Bench can create cost-effective open-source models that rival or exceed proprietary baselines in policy-oriented tasks while preserving general capabilities.
Large Language Models (LLMs) are rapidly transforming various sectors, and their potential in public governance is increasingly recognized. However, deploying these powerful AI tools in governmental scenarios comes with unique challenges, especially concerning accuracy, compliance, and the ability to handle complex, bilingual policy documents. A new research paper introduces POLIS-Bench, a groundbreaking evaluation suite designed to rigorously assess LLMs for these specific governmental bilingual policy tasks.
The paper, titled “POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarios,” was authored by Tingyue Yang, Junchi Yao, Yuhui Guo, and Chang Liu. It highlights the critical need for a systematic and robust evaluation framework to prevent issues like “false success,” where an AI-generated response appears plausible but violates policy intent or compliance rules.
Addressing the Gaps in LLM Evaluation for Governance
Existing benchmarks often fall short in evaluating LLMs for policy-oriented generation. They typically have small sample sizes, lack diverse task categories, and rely on inefficient manual scoring. POLIS-Bench addresses these limitations with three major advancements:
-
Up-to-date Bilingual Corpus: It features an extensive and current policy corpus, aggregating recent Chinese and English policy texts from official government sources. This ensures the evaluation is relevant to contemporary governance practices.
-
Scenario-Grounded Task Design: The benchmark distills three specialized, real-world tasks from policy documents: Clause Retrieval & Interpretation, Solution Generation for concrete problems, and Compliance Judgment. These tasks comprehensively test an LLM’s understanding and application of policy.
-
Dual-Metric Evaluation Framework: POLIS-Bench employs a novel dual-metric system. It combines semantic similarity (measuring lexical alignment between model outputs and reference answers) with LLM.Judge accuracy (assessing genuine task correctness and adherence to compliance constraints). This provides a more precise and comprehensive evaluation than single-metric approaches.
Key Findings from Extensive Evaluation
The researchers conducted a large-scale evaluation of over 10 state-of-the-art LLMs using POLIS-Bench. The results revealed several important insights:
-
Reasoning Models Excel: Models designed for reasoning consistently outperformed chat-based models across tasks, demonstrating superior cross-task stability and accuracy. This suggests that governmental tasks, which often require complex logical chains from comprehension to reasoning, benefit significantly from reasoning capabilities.
-
Language Performance Differences: Most models performed better when handling English policies compared to Chinese policies. This indicates a need for further optimization in multilingual adaptation, particularly for Chinese policy tasks.
-
DeepSeek-R1’s Chinese Prowess: Notably, the open-source reasoning model DeepSeek-R1 showed strong performance in Chinese policy tasks, achieving the highest accuracy rate in this category.
-
Compliance Judgment is the Hardest: Across all models and languages, the Compliance Judgment tasks proved to be the most challenging. This is likely due to the need for models to process extensive contextual information and perform complex reasoning to identify precise constraint boundaries.
Also Read:
- Beyond Universal Safety: Stress-Testing LLMs with Custom Behavioral Policies
- Unifying Software Engineering Evaluation for AI Coding Agents with SWE-Compass
Cost-Efficient Task Adaptation with POLIS Series Models
Beyond evaluation, POLIS-Bench also serves as a tool for improving LLMs. By fine-tuning lightweight open-source models (Qwen3-8B and DeepSeek-R1-distill-llama-8b) on the POLIS-Bench dataset, the researchers developed the “POLIS series” models. These fine-tuned models achieved significant improvements in accuracy, in some cases matching or even surpassing strong proprietary baselines like the GPT-4 series, while maintaining smaller parameter sizes and lower operational costs.
The fine-tuning process effectively mitigated common LLM issues such as factual hallucination (fabricating non-existent clauses) and redundant generation, leading to more accurate, concise, and policy-aligned responses. Crucially, a separate evaluation on the GPQA-Diamond benchmark confirmed that this task-aligned fine-tuning did not compromise the models’ general reasoning and knowledge retrieval abilities.
In conclusion, POLIS-Bench offers a robust and systematic framework for evaluating and enhancing LLMs for the complex demands of governmental bilingual policy tasks. It paves the way for more reliable, compliant, and cost-effective AI deployments in public governance. For more details, you can read the full research paper here.
