TLDR: GovRelBench is a novel benchmark designed to evaluate how well Large Language Models (LLMs) understand and generate content relevant to the government domain. It introduces GovRelBERT, an evaluation tool powered by the SoftGovScore method, which converts traditional labels into nuanced ‘soft scores’ to quantify relevance. This approach addresses the current gap in LLM evaluation, which often overlooks core domain capabilities in favor of safety, and has shown superior performance compared to existing methods and even much larger LLMs.
Large Language Models (LLMs) have transformed the field of Natural Language Processing, with powerful models like ChatGPT, Claude, LLaMA, and Qwen rapidly advancing. These models are increasingly being customized for specific sectors such as finance, healthcare, and law. However, when it comes to the government domain, evaluating these models presents unique challenges.
Current evaluations of LLMs in government primarily focus on safety in specific scenarios. There’s a significant gap in assessing their core capabilities, especially their relevance to government-specific topics. Government knowledge often lacks universal applicability, as standards can vary greatly across countries and regions. To address this, a new benchmark called GovRelBench has been introduced.
Introducing GovRelBench and GovRelBERT
GovRelBench is specifically designed to evaluate the core capabilities of LLMs in the government domain. It consists of a set of government-related prompts and a dedicated evaluation tool named GovRelBERT. The process is straightforward: prompts from GovRelBench are given to an LLM, and then GovRelBERT assesses the relevance of the LLM’s generated output to the government domain. This assessment provides a quantitative measure of the LLM’s performance in this specialized area.
At the heart of GovRelBERT’s training is a novel method called SoftGovScore. Initially, classifying text as simply ‘governmental’ or ‘non-governmental’ proved ineffective due to the fuzzy boundaries and overlaps with other domains, like news. For example, government activities reported online often fall into the news category, showing strong relevance, while documents from specific departments might relate to both governance and their respective fields, showing weaker relevance. Recognizing that relevance is a spectrum rather than a binary choice, SoftGovScore converts traditional ‘hard labels’ (like 1 for government, 0 for other) into ‘soft scores’ or continuous probabilities. This allows the model to learn finer-grained relevance measures, providing a more accurate quantification of a text’s connection to the government domain.
GovRelBERT itself is built upon the ModernBERT architecture. ModernBERT was chosen for its ability to handle long texts (up to 8192 tokens, compared to BERT’s 512) and its efficient inference capabilities, making it suitable for real-world deployment. The training of GovRelBERT involved a dataset created from self-crawled data combined with specific domain data filtered from open-source datasets, all processed using the SoftGovScore method.
Also Read:
- A New Milestone in Vietnamese Legal AI: Introducing the VLQA Dataset
- Automating GitHub README Classification with Large Language Models
Performance and Impact
Experiments showed that GovRelBERT significantly outperforms traditional machine learning methods and even much larger decoder-only LLMs in identifying government domain relevance. For instance, GovRelBERT demonstrated superior performance compared to a 32B LLM, which has approximately 200 times more parameters. This highlights GovRelBERT’s efficiency and effectiveness in this specific task, especially considering that LLMs were tasked with classification, while GovRelBERT performed a more nuanced regression-based scoring.
The benchmark has been applied to evaluate several prominent LLMs, including DeepSeek-Chat, Qwen1.5-72B-Instruct, and GPT-4o. The results provide insights into how well these general-purpose LLMs perform on Chinese governmental domain tasks. For example, Qwen/Qwen1.5-72B-Instruct showed strong performance among models primarily developed in China, while GPT-4o exhibited a notably better understanding of Chinese governmental context compared to Claude-3 Opus.
While GovRelBench and GovRelBERT offer a significant step forward, the researchers acknowledge limitations, such as the subjective judgment involved in data preparation and the inherent noise in large-scale source datasets. Future work aims to expand the generalizability of SoftGovScore to other specialized domains and develop more objective mechanisms for quantifying relevance.
This work provides an effective tool for relevant research and practice, enhancing the capability evaluation framework for large models in the government domain. The code and dataset for GovRelBench are openly available for the community to use and contribute to. You can find more details in the research paper itself. Read the full paper here.
