TLDR: ACTOR is a new, efficient training framework that reduces over-refusal in Large Language Models (LLMs) by precisely adjusting internal activation patterns in a single model layer. It uses “just enough” individualized shifts based on a “refusal vector” to make LLMs more helpful on benign prompts while maintaining safety on harmful ones, all with less data and computation than traditional methods.
Large Language Models (LLMs) are incredibly powerful, but sometimes their safety features can be a bit too cautious. This often leads to what’s called “over-refusal,” where the LLM rejects perfectly harmless questions because they might sound a little bit like something dangerous. Imagine asking a model for a recipe for a “killer chili,” and it refuses because it thinks you’re asking how to harm someone. This is a common problem that frustrates users and limits how useful these models can be.
Researchers have been trying to fix this issue, but many existing methods have their own drawbacks. Some training methods require a lot of data and computing power, making them expensive and slow. Other methods that try to fix the problem during the model’s response generation (inference-time) can be computationally heavy or not very stable when the types of questions change.
A new approach called ACTOR (ACtivation-Based Training for Over-Refusal Reduction) offers a promising solution. This framework is designed to be robust, and efficient in terms of both computing power and data. ACTOR tackles over-refusal by looking at the internal “thought patterns” or “activations” of the LLM when it processes different types of questions. It precisely identifies and adjusts the specific parts of these internal patterns that cause the model to refuse a request.
The core idea behind ACTOR is to fine-tune only a single layer of the LLM. This is a significant departure from methods that retrain large parts of the model. By focusing on a single layer, ACTOR gains very precise control over the refusal mechanism. It works by first identifying a “refusal vector” – essentially, a direction in the model’s internal representation space that indicates whether a query is likely to be refused or answered. This vector helps ACTOR understand how the model’s internal state shifts when it moves from understanding a benign query to a harmful one.
Instead of applying a uniform adjustment to all queries, ACTOR uses “individualized shifts.” This means it calibrates how much to adjust the model’s internal state for each question based on how “close” that question is to triggering a refusal. For questions that are only slightly similar to harmful ones (pseudo-harmful), it applies just enough of a shift to make the model respond appropriately without over-correcting. For genuinely harmful questions, it strengthens the refusal mechanism.
This “just enough” shift is estimated efficiently by looking at how much a query’s internal representation aligns with the refusal vector. If a query’s representation strongly points towards the refusal direction, it needs a larger correction. This method avoids the computational expense of generating and judging responses during training, making it much faster and more data-efficient.
Experiments show that ACTOR significantly improves the “Compliance Rate” (how often the model gives a proper answer instead of refusing) across various benchmarks for models like Llama-2-7b-chat, Llama-2-13b-chat, and Gemma-7b-it. For instance, it boosted the average Compliance Rate for Llama-2-7b-chat by over 47%. Crucially, it does this while maintaining the model’s ability to correctly refuse genuinely harmful queries, ensuring safety isn’t compromised. The “Safety Score” on harmful benchmarks remained very high.
One of ACTOR’s standout features is its efficiency. Training with ACTOR takes significantly less time compared to traditional Supervised Fine-Tuning (SFT) – for example, only 4 minutes versus 15 minutes for Llama-2-7b-chat. It also requires less data, performing well even with a small number of pseudo-harmful examples. Furthermore, ACTOR is robust to changes in the data used to calculate the refusal direction, unlike some other methods that can be quite sensitive.
Also Read:
- Streamlining LLM Reasoning: A New Approach to Chain-of-Thought Compression
- Uncovering ‘Attention Slipping’: A New Insight into LLM Jailbreaks and a Novel Defense
The impact on the model’s general capabilities, such as its ability to follow instructions or answer knowledge-based questions, is minimal. This means that while ACTOR makes LLMs less prone to over-refusal, it doesn’t degrade their overall performance. This work offers a promising path forward for making LLMs more helpful and reliable for users without sacrificing crucial safety measures. You can find more details in the research paper.
