TLDR: This research assesses Vision-Language Models (VLMs) like Qwen and Gemini for classifying radio galaxies (FR-I/FR-II) using the MiraBest dataset. It finds that while prompt-based approaches can perform well, VLM outputs are highly sensitive to minor prompt changes. However, with lightweight LoRA fine-tuning (15M parameters), generic VLMs can achieve near state-of-the-art performance (3% error), rivaling specialized models, suggesting they are promising but fragile tools for scientific discovery requiring careful prompt design and adaptation.
Vision-Language Models (VLMs) like Qwen and Gemini are powerful AI systems designed to understand and reason across different types of data, including images and text. While they excel in general tasks, their effectiveness in specialized scientific fields, particularly with unfamiliar datasets like those found in astronomy, has been less clear. A recent research paper explores this very question, focusing on how well generic VLMs can classify radio galaxies and what strategies work best to improve their performance.
Understanding Radio Galaxies with AI
The study, titled “Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation”, delves into the challenge of classifying radio galaxies into two main types: Fanaroff–Riley Type I (FR-I) and Type II (FR-II). FR-I galaxies typically have bright central cores with jets that fade as they extend, while FR-II galaxies show edge-brightened lobes with prominent hotspots at their ends. This classification is crucial for astronomers, and the researchers used the MiraBestFR-I/FR-II dataset, a collection of radio images labeled by experts, for their assessment.
Prompting Strategies and Model Adaptation
The core of the research involved testing various ways to “prompt” these AI models. The team explored several strategies:
Natural Language Descriptions: Providing text-based definitions of FR-I and FR-II galaxies.
Schematic Diagrams: Augmenting text descriptions with abstract visual diagrams illustrating the galaxy types.
Visual In-Context Examples: Introducing labeled support images directly into the prompts, a novel approach in astronomy for VLMs. This included using fixed sets of images or dynamically retrieved nearest neighbors (kNN-Imgs) based on visual similarity.
Beyond prompting, the researchers also evaluated a lightweight supervised adaptation technique called LoRA (Low-Rank Adaptation). This method fine-tunes the VLM with a small number of trainable parameters (around 15 million) without requiring extensive astronomy-specific pre-training.
Key Findings: Promise and Fragility
The study revealed several important trends:
Firstly, even basic prompt-based approaches showed good performance. This suggests that general-purpose VLMs already possess a foundational understanding that can be useful for unfamiliar scientific domains, even without prior exposure to astronomical data.
Secondly, a significant finding was the high instability of the model outputs. Minor changes to the prompt, such as altering the layout, the order of examples, or even the decoding temperature (which controls the randomness of the output), could drastically change the results. This indicates that the apparent “reasoning” of VLMs might often be a reflection of their sensitivity to prompt construction rather than deep, genuine inference.
Thirdly, the lightweight adaptation via LoRA fine-tuning proved remarkably effective. With just 15 million trainable parameters and no specialized astronomy pre-training, a fine-tuned Qwen-VL model achieved a near state-of-the-art error rate of 3%. This performance rivals that of domain-specific models that are extensively pre-trained on astronomical data, highlighting the potential of generic VLMs as powerful, data-efficient tools for scientific discovery, provided they are properly adapted.
For instance, Gemini models performed strongly in zero-shot settings (without examples), achieving errors as low as 14% with just text prompts. Open-source models like Qwen improved significantly when conditioned on retrieved visual examples. However, the study also noted that Chain-of-Thought (CoT) prompting, which asks models to explain their reasoning, generally increased variance and often led to worse performance, suggesting that while it has potential, its effective use requires careful supervision.
Also Read:
- New Benchmark Challenges AI’s Understanding of Space
- Waste-Bench: A New Benchmark Reveals VLLM Challenges in Real-World Waste Classification
Implications for Scientific Discovery
The research concludes that while Vision-Language Models hold immense promise for scientific imaging, particularly in fields like radio astronomy, their application requires a nuanced approach. Their success is critically dependent on how prompts are constructed and the adaptation methods used. The ability of generic VLMs to achieve high performance with minimal fine-tuning is a significant step forward, offering a scalable and data-efficient alternative to building specialized models from scratch. However, the observed prompt sensitivity underscores the need for caution and rigorous testing when deploying these models in critical scientific applications.
For more in-depth information, you can read the full research paper here.
