TLDR: Large Language Models (LLMs), despite being trained only on text, develop inherent visual knowledge, or “visual priors.” This research systematically analyzes these priors, finding they consist of separable perception and reasoning abilities. Reasoning priors are cultivated by data like code and math, scaling progressively and being universally applicable to visual reasoning. Perception priors emerge from diverse text and are more dependent on visual encoders and instruction tuning. The study proposes a data-centric strategy to pre-train LLMs that are more “vision-aware,” leading to stronger multimodal performance, validated at a 1T token scale.
Large Language Models (LLMs), traditionally trained solely on vast amounts of text, have shown a surprising ability to develop a rich understanding of the visual world. This inherent visual knowledge, termed ‘visual priors,’ allows these models to perform vision tasks with minimal additional visual data, and sometimes even without ever having seen an image. A recent research paper, “Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training,” delves deep into this fascinating phenomenon, systematically analyzing how these visual priors are formed and how they can be deliberately cultivated.
The researchers, Junlin Han, Shengbang Tong, David Fan, Yufan Ren, Koustuv Sinha, Philip Torr, and Filippos Kokkinos, from Meta Superintelligence Labs and the University of Oxford, reveal that these visual priors are not a single, uniform block of knowledge. Instead, they are composed of two distinct and separable components: a perception prior and a reasoning prior. Each of these priors exhibits unique scaling trends and originates from different types of language pre-training data.
The Origins of Visual Reasoning
The study highlights that an LLM’s latent visual reasoning ability is predominantly developed by pre-training on ‘reasoning-centric’ data. This includes categories such as code, mathematics, and academic texts. As the model is exposed to more of this type of data, its reasoning prior scales progressively, becoming stronger and more refined. Crucially, this reasoning ability acquired from language pre-training is highly transferable and universally applicable to various visual reasoning tasks. This suggests that the model learns abstract principles of logic and structure from text, which it can then apply to understand and interpret visual information.
The Nature of Visual Perception
In contrast to reasoning, the perception prior emerges more diffusely from broad and diverse language corpora. This means that a wide variety of text, rather than specific categories, contributes to the model’s basic perceptual understanding. The research found that while text describing the visual world is important, its impact on performance saturates rapidly. Furthermore, the model’s perception ability is more sensitive to the characteristics of the vision encoder (the component that processes images) and the visual instruction tuning data used during the multimodal adaptation phase.
A Data-Centric Recipe for Vision-Aware LLMs
Leveraging these insights, the researchers propose a practical, data-centric recipe for pre-training LLMs that are inherently more ‘vision-aware.’ Their findings, grounded in over 100 controlled experiments consuming 500,000 GPU-hours, demonstrate that a carefully calibrated data mixture can cultivate powerful visual priors without significantly compromising core language abilities. They identified an optimal blend that is heavily skewed towards reasoning-centric content but also includes the necessary visual world knowledge.
The study also introduces two valuable resources for the multimodal LLM community: the Multi-Level Existence Bench (MLE-Bench), designed for fine-grained evaluation of perceptual abilities, and a ‘blind visual instruction tuning’ trick, which can improve visual adaptation by initially training on text-only instructions.
Also Read:
- Decoding How AI Understands the World: A Multimodal Perspective
- Improving Robot Navigation with Contextual Textual Descriptions in LLMs
Scaling Up and Future Directions
The principles identified in smaller-scale studies were validated by scaling up the approach to train 7B parameter LLMs on 1 trillion tokens. The ‘balanced’ model, pre-trained with the proposed vision-aware data mixture, consistently outperformed a ‘language-favorable’ model on visual tasks while maintaining competitive language proficiency. This confirms that deliberate pre-training choices can imbue LLMs with stronger visual priors at a larger scale.
This work provides a new way of deliberately cultivating visual priors from language pre-training, paving the way for the next generation of multimodal LLMs. It offers strong empirical support for the “Platonic Representation Hypothesis,” which posits that powerful models can learn a unified internal world model from any single modality, such as text. For more in-depth information, you can read the full research paper here.
