TLDR: L2S (Learn-to-Steer) is a new method for guiding Multimodal LLMs (MLLMs) to produce safer and more accurate responses. Unlike previous methods that use a fixed steering approach, L2S learns to apply input-specific adjustments using a small auxiliary network. This allows MLLMs to adapt their behavior based on the input context, effectively reducing hallucinations and enforcing safety with minimal computational cost.
Large Language Models (LLMs) that can understand and generate content across different modalities, like text and images, are becoming increasingly common. These Multimodal LLMs (MLLMs) are powerful, but they still face significant challenges, particularly when it comes to generating accurate and safe responses. Two major issues are “hallucinations,” where the model outputs information not grounded in its input, and “safety concerns,” where it might provide harmful or inappropriate content.
Traditionally, addressing these issues often involves fine-tuning the entire model, which can be very expensive and time-consuming due to the ever-growing size of MLLMs. A more cost-effective alternative that has gained traction is “model steering.” This technique works by making small, linear adjustments to the model’s internal representations, guiding its output towards a desired behavior. However, most existing steering methods, like “mean steering,” use a single, fixed adjustment regardless of the specific input query. This static approach limits their effectiveness, as the “correct” or “safe” behavior can vary greatly depending on the context of the input.
For instance, if an MLLM is asked about an illegal activity, a safe response would be to refuse to engage. But if asked for financial advice, a safe response would be to suggest consulting an expert, not to refuse entirely. Recognizing this limitation, a new approach called Learn-to-Steer (L2S) has been developed. L2S introduces a more nuanced, input-dependent steering mechanism.
The core idea behind L2S is inspired by “Prompt-to-Steer” (P2S). P2S generates a unique steering adjustment for each input by contrasting desired and undesired behaviors using specific prompts. While P2S is effective, it’s not practical for real-world use because it requires knowing these specific prompts at the time of inference. L2S solves this by training a small, lightweight auxiliary network. This network learns to predict the appropriate input-specific steering adjustment based on the input’s internal representation, without needing the explicit contrastive prompts at inference time.
The auxiliary network in L2S is remarkably efficient, requiring minimal computational resources for training and operation. This means L2S can provide expressive, input-dependent behavior corrections without the heavy overhead of fine-tuning. The researchers demonstrated the effectiveness of L2S in two critical applications: reducing hallucinations and enforcing safety in MLLMs. In experiments, L2S significantly outperformed traditional, input-independent steering methods.
For safety enforcement, L2S was tested on the MMSafetyBench dataset, which includes multimodal queries designed to elicit unsafe responses. L2S showed a notable reduction in harmful outputs and a better ability to defer to human experts when appropriate, compared to other baselines. In mitigating hallucinations, evaluated on the POPE dataset and COCO validation set, L2S consistently reduced the generation of ungrounded content, leading to more accurate and coherent responses. The model also showed a higher preference rate in human evaluations for its descriptive quality.
Also Read:
- Enhancing MLLM Accuracy: A New Method for Controlled Image Captioning
- Language Models Learn to Think and Refine on the Fly
This innovative approach marks a significant step forward in making MLLMs more reliable and trustworthy by enabling fine-grained, context-aware control over their behavior. For more technical details, you can refer to the full research paper here.
