TLDR: A new research paper introduces the concept of “self-contradiction” in Multimodal Large Language Models (MLLMs), where their image generation doesn’t align with their own understanding. The study found this is mainly due to weak generation, not misunderstanding. By using the model’s stronger understanding branch to guide its weaker generation branch, a “co-improvement” effect is observed, enhancing both capabilities without external supervision. However, poor data quality can lead to “co-degradation,” which internal metrics can’t detect. A curriculum-based learning strategy is proposed to leverage co-improvement, leading to more unified and capable MLLMs.
Multimodal Large Language Models, or MLLMs, are designed to handle both image understanding and generation tasks. Ideally, these models should be perfectly unified, meaning what they generate aligns perfectly with what they understand from a prompt. However, a recent research paper highlights a fascinating and problematic phenomenon: MLLMs often contradict themselves. This means that an image generated by the model might be deemed misaligned with the original prompt by the model’s own understanding component.
This internal inconsistency, termed “self-contradiction,” reveals a significant gap between the generation and understanding capabilities within these advanced AI models. Researchers quantified this issue using a “Nonunified score,” which measures the proportion of times the understanding branch judges a generated image as misaligned with the input prompt. A perfect MLLM would have a score of zero, but current models show a clear non-zero score, indicating a lack of true unification.
Upon deeper investigation, the study found that this self-contradiction primarily stems from a “weak generation” capability. In simpler terms, the model struggles to produce images that accurately reflect the prompt, rather than misunderstanding the prompt itself. The understanding branch, in most cases, correctly identifies these misaligned generations. This asymmetry—where understanding is stronger than generation—presents a unique opportunity for self-improvement.
The core idea proposed by the researchers is to leverage the stronger understanding branch to guide and improve the weaker generation branch. This approach allows MLLMs to enhance themselves without needing any external human feedback or additional datasets. They applied standard machine learning techniques like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), treating the model’s own understanding branch as an internal reward system.
A remarkable discovery from this research is the “co-improvement effect.” When the generation branch is fine-tuned using this internal supervision, not only does its image generation quality improve, but the model’s understanding capabilities are also enhanced simultaneously. This phenomenon, while observed in earlier stages of model training, was largely unexplored in post-training. The improvement in understanding primarily comes from the model’s increased ability to correctly identify “false positives”—images it previously thought were aligned with the prompt but were actually not.
However, the study also uncovered a potential pitfall: “co-degradation.” If the model is trained with low-quality or corrupted data, both its generation and understanding capabilities can decline together. This highlights a critical limitation: internal metrics like the Nonunified score, while useful for measuring unification, cannot distinguish between genuine co-improvement and harmful co-degradation. This underscores the necessity of ensuring data quality, even when using internal feedback.
To further harness the co-improvement effect, the researchers propose a curriculum-based learning strategy. This method gradually introduces more challenging samples for training as the model’s capabilities evolve. By starting with easier examples and progressively moving to harder ones, the model can dynamically expand its training data, leading to better overall performance and unification. Experiments showed that this curriculum-based approach consistently improved both generation and understanding, and further reduced the Nonunified score across various tasks.
Also Read:
- When Visual Input Challenges VLM’s Internal Facts
- Enhancing Multimodal AI: A New Approach to Visual Understanding in Context
This research offers a novel perspective on how MLLMs can achieve self-improvement by addressing their internal inconsistencies. It suggests a path towards more unified and capable multimodal AI models, relying on their own inherent strengths. For more technical details, you can refer to the full research paper: Self-Contradiction as Self-Improvement: Mitigating the Generation-Understanding Gap in MLLMs.
