TLDR: CLARITY is a new reinforcement learning framework that improves the reasoning quality and accuracy of expert LLMs in data-scarce domains like law and medicine. It uses a small, general-purpose LLM to monitor reasoning consistency, integrating a two-stage ‘refine-then-monitor’ training pipeline and dynamic data reformulation. This approach prevents the degradation of logical consistency often seen with standard outcome-based RL on multiple-choice questions, leading to more coherent, professional, and readable responses without relying on expensive large models or extensive expert annotations.
In the rapidly evolving field of artificial intelligence, training large language models (LLMs) to be experts in specialized domains like law and medicine presents unique challenges. These domains often suffer from a scarcity of high-quality, diverse training data, frequently relying on multiple-choice questions (MCQs) for evaluation. While outcome-based reinforcement learning (RL) on MCQs can boost accuracy, a recent study highlights a critical flaw: it often degrades the reasoning quality of LLMs, leading to logical inconsistencies.
A new research paper introduces CLARITY, a cost-effective reinforcement learning framework designed to tackle this very issue. CLARITY aims to enhance the reasoning quality of expert LLMs using only a small, general-purpose LLM, sidestepping the need for expensive, large-scale Process Reward Models (PRMs) or extensive expert-annotated datasets. This innovative approach integrates a consistency-aware reward mechanism with a two-stage ‘refine-then-monitor’ training pipeline and a dynamic data reformulation strategy to maximize the utility of limited data.
The core problem identified by the researchers is that standard RL, when applied to MCQs, tends to reward only the final correct answer. This can lead models to find shortcuts or even guess correctly without developing robust, logically sound reasoning processes. A pilot study conducted on a judicial examination MCQ dataset demonstrated this clearly: while final answer accuracy improved, the overall response quality, particularly logical consistency, significantly declined. The proportion of responses with logical fallacies rose from 7% to 31%.
CLARITY addresses this by introducing a ‘consistency reward’. This mechanism evaluates the logical flow and consistency within an LLM’s reasoning process. It works by separating the model’s response into its reasoning trajectory and the final answer. A smaller, general-purpose LLM acts as a consistency reward model, identifying the options the model believes are correct based on its reasoning. A penalty is applied if the reward model cannot clearly identify these judgments or if the believed-correct options don’t match the final answer. This encourages the model to generate more coherent and reliable reasoning.
A significant advantage of this consistency-aware mechanism is its minimal requirement for domain expertise. The reward model only needs to understand basic correctness judgments (e.g., ‘Option A is correct’) rather than deep domain knowledge, making it feasible to use smaller, general-purpose LLMs. Despite this narrow focus on logical coherence, the researchers observed broader improvements in overall reasoning proficiency.
The framework also includes a two-stage ‘refine-then-monitor’ training pipeline. Stage 1 focuses on refining the model’s output structure, encouraging an option-by-option reasoning format. This makes the reasoning process transparent and easier for the consistency checker to evaluate, preventing ‘reward hacking’ where models might simplify reasoning to avoid penalties. Stage 2 then relaxes these format constraints, using the consistency reward model to monitor deeper reasoning while also incorporating an answer reward to optimize for correctness. This stage also employs a strict reward mechanism, giving positive feedback only when all correct options are selected, pushing the model towards more profound reasoning.
To combat data scarcity, CLARITY uses a dynamic data reformulation strategy. This involves deconstructing existing MCQ instances into independent propositions, polishing their phrasing, and diversifying them with fictional names and places. During training, easier instances are dynamically reformulated into more challenging ones by randomly grouping these propositions, effectively creating more diverse and difficult training data without needing additional expert annotations.
Also Read:
- Beyond Surface Metrics: Detecting Data Contamination in LLMs with Internal Analysis
- The Locality Dial: Bridging Interpretability and Performance in LLMs
Experimental results show that CLARITY significantly improves response consistency by 16.5% and reliable reasoning accuracy by 7.5% over standard RL baselines. It also demonstrates strong generalizability across various unseen open-ended tasks and alternative MCQ formats. Human evaluations further confirm that CLARITY-trained models exhibit holistic improvements in coherence, professionalism, and readability, sometimes even surpassing large commercial systems like GPT-4o. This suggests that smaller, general-purpose LLMs can indeed effectively guide the training of expert models by focusing on reasoning consistency. For more details, you can refer to the full research paper here.
