TLDR: A new research paper introduces an efficient method to estimate how difficult a question is for a Large Language Model (LLM) by analyzing its internal ‘hidden representations’ rather than its generated outputs. By modeling the LLM’s thinking process as a Markov chain and using a value function, the approach accurately predicts question difficulty based on the initial input, without generating any tokens. This leads to faster and more accurate difficulty assessment, enabling LLMs to adapt their reasoning strategies more efficiently and preserve their general capabilities, outperforming existing methods across diverse tasks and datasets.
Large language models (LLMs) are becoming increasingly powerful, and understanding how difficult a question is for an LLM to answer is crucial. This understanding can help us evaluate their performance more accurately, train them better, and even make their inference processes more efficient. Imagine an LLM that knows when a question is tough for it; it could then decide to think harder or use more sophisticated strategies, saving computational resources on easier questions.
Current methods for estimating question difficulty often come with significant drawbacks. Some approaches involve generating multiple responses to see how consistent the LLM’s answers are, which can be very costly in terms of computation. Others rely on separate, auxiliary models to judge difficulty, or even require fine-tuning the target LLM itself, which might inadvertently reduce its overall capabilities or make it less robust.
A new research paper, titled “The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations,” introduces a novel and efficient way to tackle this challenge. Instead of looking at the LLM’s final output or requiring extensive additional processes, this method taps into the model’s internal “thoughts” – specifically, its hidden representations. These hidden representations are the intermediate data structures within the LLM that capture its understanding and reasoning process as it processes an input.
The core idea is quite intuitive: if an LLM finds a question difficult, this difficulty should be reflected in its internal processing, not just its final answer. The researchers found that these hidden representations can clearly distinguish between easy and hard questions, suggesting they hold valuable information about perceived difficulty.
The proposed method models the LLM’s token-by-token generation process as a Markov chain. In simple terms, this means that each step in the LLM’s thinking process (represented by a hidden state) influences the next. To quantify difficulty, a “value function” is defined over this Markov chain. This function estimates the expected quality of the LLM’s output, given any internal hidden state. Crucially, this allows for an accurate estimation of difficulty based solely on the initial hidden state, which is derived directly from the input question, without needing to generate any output tokens at all. This makes the process incredibly efficient.
The benefits of this approach are significant. It offers accurate difficulty estimation without the computational overhead of generating multiple outputs. It leverages the LLM’s own internal signals, providing a more direct reflection of its reasoning. Furthermore, because it doesn’t involve fine-tuning the target LLM, it preserves the model’s general capabilities, ensuring it remains robust and safe.
The researchers conducted extensive experiments across various tasks, including general-purpose reasoning, mathematical reasoning, and open-ended problem-solving. These tasks covered both purely text-based and multimodal (image-text) scenarios, using datasets like MMBench, ScienceQA, MathVista, StrategyQA, GSM8K, and CommonsenseQA. They tested their method with advanced open-source multimodal LLMs such as Qwen2.5-VL-7B-Instruct and InternVL3-8B.
The results were compelling. The new method consistently outperformed existing baselines in accurately classifying question difficulty, achieving high ROC-AUC and Macro-F1 scores. For instance, on the ScienceQA dataset, it reached an ROC-AUC of 93.09% and a Macro-F1 of 79.48%. Moreover, it proved to be significantly faster, requiring less time for difficulty assessment at test time compared to other methods.
Beyond just estimating difficulty, the method was also applied to guide adaptive reasoning strategies like Self-Consistency, Best-of-N, and Self-Refine. By tailoring the reasoning effort based on the estimated difficulty – using simpler strategies for easy questions and more complex ones for hard questions – the approach achieved higher inference efficiency with fewer generated tokens while maintaining or improving accuracy. This means LLMs can be smarter about how they allocate their computational resources.
Further analysis demonstrated the method’s robustness and generalizability across open-ended questions, different datasets, and even various model sizes within the same architecture family. While direct generalization across entirely different LLM architectures remains a challenge, the approach allows for training a lightweight value function for each target model, which is still much faster than other training-based methods.
Also Read:
- Unveiling LLM Self-Awareness: Predicting Answer Accuracy Before Generation
- Mapping LLM Reasoning: A Graph-Based Approach to Confidence Estimation
In conclusion, this research presents a lightweight and highly effective approach for estimating question difficulty in LLMs by leveraging their hidden representations. By modeling the generation process as a Markov chain and using a value function over hidden states, it enables efficient and accurate difficulty estimation without needing to generate outputs. This not only improves difficulty classification but also enhances inference efficiency in adaptive reasoning scenarios. For more details, you can read the full paper here.
