TLDR: CRACQ is a new multi-dimensional framework for automated document assessment, evaluating text across five traits: Coherence, Rigor, Appropriateness, Completeness, and Quality. It uses an LLM-as-a-judge to create a synthetic dataset for training a BigBird RoBERTa model, fine-tuned with LoRA. CRACQ provides stable, interpretable trait-level judgments, offering detailed feedback beyond single-score approaches, and has been released publicly for further research.
In the evolving landscape of AI-generated content, accurately assessing the quality of open-ended documents like grant proposals or long-form articles remains a significant challenge. Traditional methods often fall short, either by providing opaque, non-deterministic scores or by offering only a single, holistic quality metric without detailed feedback. To address these limitations, researchers from Granter.ai have introduced CRACQ, a novel multi-dimensional evaluation framework designed for automated document assessment. You can read the full research paper here: CRACQ: A Multi-Dimensional Approach To Automated Document Assessment.
Understanding CRACQ’s Core Dimensions
CRACQ stands for Coherence, Rigor, Appropriateness, Completeness, and Quality. These five specific traits form the backbone of its evaluation system, moving beyond a single score to provide a nuanced, rubric-driven assessment. Each dimension focuses on a critical aspect of document quality: Coherence evaluates the logical consistency and flow of the narrative; Rigor assesses the thoroughness and soundness of arguments presented; Appropriateness determines how well the document fits the specific goals and intended audience; Completeness checks if all required elements and objectives are addressed without critical gaps; and Quality judges the overall writing quality and persuasiveness of the document.
How CRACQ Works: A Glimpse into the Methodology
To train CRACQ, the team at Granter.ai created a unique dataset of 500 synthetic grant proposals. These proposals were generated using an AI-driven platform and validated by human grant experts to ensure they mirrored real-world applications. An LLM-as-a-judge (LAAJ) system was then employed to evaluate each synthetic proposal across the five CRACQ dimensions. This LAAJ system was guided by specific, rubric-anchored questions for each trait, ensuring its judgments were consistent and interpretable.
The evaluation involved a two-pass procedure: first, a structured assessment using 25 trait-specific questions (five per CRACQ dimension), where the LAAJ provided a numeric score and a brief rationale. Second, these scores were aggregated to produce a mean score for each dimension and an overall CRACQ score.
For the automated evaluator, CRACQ utilizes a BigBird RoBERTa architecture, known for its ability to handle long documents efficiently. This pre-trained model was fine-tuned using Low-Rank Adaptation (LoRA), a technique that significantly reduces the computational resources needed while maintaining high performance. The model was trained to predict scores for each trait independently, further enhancing interpretability.
Performance and Interpretability
Preliminary results indicate that CRACQ offers more stable and interpretable trait-level judgments compared to direct LLM evaluations. When benchmarked against the LAAJ system and tested on real grant applications, CRACQ demonstrated strong performance. For instance, Appropriateness consistently showed the best alignment with reference judgments, exhibiting the lowest error and highest correlation. Coherence and Quality also performed strongly, while Completeness proved to be the most challenging dimension to assess accurately.
The framework’s ability to provide a breakdown across distinct dimensions—rather than a single “black-box” score—is a key advantage. This detailed feedback can help users diagnose specific areas for improvement in their documents, making the assessment actionable.
Also Read:
- Evaluating RAG Systems with Knowledge Graphs: A New Framework for Deeper Semantic Analysis
- How Well Do LLMs Tutor? A New Benchmark Reveals Strengths and Weaknesses
Looking Ahead
CRACQ represents a significant step towards more sophisticated automated document evaluation. The researchers have publicly released the CRACQ model and framework, encouraging further research and development. Future work aims to incorporate more diverse training data, including human-labeled examples, and explore advanced architectural methods to enhance trait-specific sensitivity, ultimately striving to narrow the gap between automated predictions and human expert judgments.
