TLDR: A research paper investigated whether increasing prompt specificity improves Large Language Model (LLM) evaluation consistency, similar to human behavior. The study found that contrary to expectations, performance did not consistently improve with more specific prompts, and performance variance remained largely unchanged. This challenges the assumption that LLMs behave like human evaluators and highlights the need for greater robustness and improved input integration in future model development, especially for high-stakes tasks like educational assessment.
Many of us interact with AI tools like ChatGPT, often thinking of them as human-like assistants. This ‘Model H’ mental model suggests that if we give more specific instructions, the AI should perform more consistently, much like a human would according to goal-setting theory.
However, a recent research paper, “The Limits of Goal-Setting Theory in LLM-Driven Assessment” by Mrityunjay Kumar, challenges this very assumption. The study investigated whether increasing the specificity of prompts given to an LLM (ChatGPT) would lead to more consistent evaluation behavior, especially in an educational assessment context.
The Experiment: Testing Prompt Specificity
The researcher conducted a controlled experiment where ChatGPT evaluated 29 student submissions. Four different prompt types were used, each with increasing levels of specificity:
- Prompt Type 1 (P1): Basic instructions, no rubric or model solution.
- Prompt Type 2 (P2): Included a model solution, but no rubric.
- Prompt Type 3 (P3): Included a rubric, but no model solution.
- Prompt Type 4 (P4): Included both a rubric and a model solution.
Each prompt type was run six times independently, and the consistency of ChatGPT’s evaluations was measured using Cohen’s Kappa, a metric for inter-rater reliability. The study focused on how performance (mean Kappa) and performance variance (standard deviation of Kappa) changed with prompt specificity.
Surprising Results on Consistency
Contrary to what the ‘Model H’ mental model and goal-setting theory would predict for humans, the study found that increasing prompt specificity did not consistently improve ChatGPT’s evaluation performance. In fact, some more specific prompts led to worse results:
- Prompt Type 3 (rubric only) yielded the highest consistency.
- Prompt Type 2 (model solution only) resulted in the lowest consistency, performing significantly worse than P1 (no rubric or model) and P3.
- Even more surprisingly, combining both a rubric and a model solution (P4) did not outperform using a rubric alone (P3).
This suggests that LLMs don’t always benefit from more information in the way a human might. The paper posits that LLMs might struggle to integrate multiple sources of guidance, potentially overfitting to specific cues rather than synthesizing all provided information effectively. This highlights a structural limitation in how current LLMs process evaluative tasks.
No Change in Performance Variance
When it came to performance variance, the results were equally revealing. The study found no significant difference in the spread of Kappa scores across any of the prompt types. This means that regardless of how specific the prompt was, the variability in ChatGPT’s performance remained largely unchanged. In essence, ChatGPT did not demonstrate performance stability with increased specificity.
Also Read:
- The Hidden Impact of a Single Character on LLM Performance and Evaluation
- Unexpected AI Behavior: Impolite Prompts Boost ChatGPT-4o Accuracy
Challenging the Human-Like AI Assumption
The findings of this research paper are significant. They challenge the common assumption that large language models behave like human evaluators. The study concludes that LLMs are neither consistent (performance doesn’t reliably improve with more specificity) nor stable (variance doesn’t decrease with more specificity) in their evaluation performance.
For fields like education, where consistent and stable assessment is crucial, these results are a call for caution. They underscore the need for future LLM development to focus not just on accuracy, but also on robustness and improved input integration mechanisms, especially for users who might instinctively apply human-centric mental models to AI interactions.
