TLDR: A new research paper reveals a significant gap in language model (LM) agents: while they possess strong knowledge of potential risks, they often fail to identify these risks in real-world scenarios or prevent themselves from executing dangerous actions. This ‘awareness-execution gap’ persists even in highly capable models. To address this, researchers developed a risk verifier system, enhanced with an abstractor, which independently critiques proposed actions and significantly reduces risky behavior by leveraging the agents’ abstract risk knowledge.
Language model (LM) agents are showing immense promise for automating various real-world tasks. However, their deployment in critical scenarios raises significant safety concerns. A recent research paper, titled “LM Agents May Fail to Act on Their Own Risk Knowledge,” highlights a crucial discrepancy: while these agents often demonstrate awareness of potential risks when directly asked, they frequently fail to identify or avoid these same risks during actual task execution.
The paper identifies this as a significant gap between an LM agent’s understanding of risks and its ability to execute safely. For instance, an agent might correctly answer “Yes” if asked, “Is executing ‘sudo rm -rf /*’ dangerous?” Yet, when faced with an actual scenario requiring such an action, it might not recognize the danger or even proceed to perform it.
To systematically investigate this issue, the researchers developed a comprehensive evaluation framework. This framework assesses agent safety across three progressive dimensions: first, their basic knowledge about potential risks; second, their ability to identify these risks within specific execution sequences; and third, their actual behavior in avoiding risky actions. All three tests in this framework are derived from the same underlying execution trajectories, allowing for a controlled comparison of different safety aspects.
The evaluation revealed two critical performance gaps that resemble the ‘generator-validator gaps’ observed in other language model applications. The first is the “Knowledge-Identification Gap.” Agents showed near-perfect risk knowledge, with over 98% pass rates in knowledge tests. However, their performance dropped significantly (by over 23%) when asked to apply this knowledge to identify risks in actual, instantiated scenarios. The second is the “Identification-Execution Gap.” Even when agents could identify risks in specific trajectories, they still often executed those risky actions, with pass rates dropping to less than 26% in execution tests.
A notable finding was that simply scaling up model capabilities or using specialized reasoning models like DeepSeek-R1 did not inherently resolve these safety concerns. This suggests that increasing a model’s general intelligence or reasoning power alone isn’t sufficient to bridge the gap between knowing a risk and acting safely.
Inspired by these findings, the researchers developed a risk mitigation strategy. They introduced a “risk verifier” that independently critiques the agent’s proposed actions. This verifier, instantiated with the same base LM as the agent, assesses potential risks before execution. To further enhance its effectiveness, an “abstractor” was added. This component converts specific execution trajectories into abstract descriptions, making it easier for the verifier to identify risks by leveraging the agent’s abstract risk knowledge. This overall system achieved a significant reduction of risky action execution by 55.3% compared to agents without these safeguards.
Also Read:
- The Self-Execution Benchmark: A Deep Dive into LLMs’ Internal Predictions
- Large Language Models Struggle to Accurately Verify Code, Study Finds
In conclusion, this research provides a systematic evaluation of LM agents’ safety profiles, revealing persistent gaps between their risk awareness and safe execution abilities. The proposed risk verification framework offers a promising approach to enhance agent safety without compromising their helpfulness. For more details, you can refer to the full research paper available at arXiv:2508.13465.
