TLDR: Research introduces Promptception, a framework with 61 prompt types, to systematically evaluate Large Multimodal Model (LMM) sensitivity to prompts. Findings show proprietary LMMs are highly sensitive to phrasing, reflecting strong instruction-following, while open-source models are more stable but struggle with complex instructions. The study reveals that minor prompt variations can cause up to 15% accuracy deviations, highlighting challenges in fair LMM evaluation and leading to proposed prompting principles for more robust assessment.
Large Multimodal Models (LMMs) have made incredible strides in integrating vision and language, allowing them to perform complex reasoning tasks with both text and images or videos. However, a new research paper titled “Promptception: How Sensitive Are Large Multimodal Models to Prompts?” by Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, and Salman Khan, sheds light on a critical, yet often overlooked, challenge: the significant impact of prompt design on LMM performance.
The researchers highlight that even minor changes in how a prompt is phrased or structured can lead to substantial accuracy deviations, sometimes as much as 15%. This variability poses a serious problem for fair and transparent evaluation of LMMs, as models often report their best performance using carefully chosen, optimized prompts. To tackle this, the team introduced Promptception, a comprehensive framework designed to systematically evaluate how sensitive LMMs are to different prompts. You can read the full research paper here.
Understanding Promptception
Promptception is an extensive framework comprising 61 distinct prompt types, organized into 15 categories and 6 supercategories. These categories target various aspects of prompt formulation, from how choices are presented to the linguistic style and even the inclusion of thought processes or ethical guidance. The framework was used to evaluate 10 LMMs, ranging from smaller open-source models to powerful proprietary ones like GPT-4o and Gemini 1.5 Pro. The evaluation spanned three multiple-choice question answering (MCQA) benchmarks: MMStar (single image), MMMU-Pro (multi-image), and MVBench (video), ensuring a broad assessment across different modalities.
Key Findings on Prompt Sensitivity
The study revealed several crucial insights into how LMMs react to prompts:
- Proprietary Models vs. Open-Source Models: Proprietary models, such as GPT-4o and Gemini 1.5 Pro, were found to be more sensitive to prompt phrasing. This suggests they are highly aligned with instruction semantics, meaning they follow instructions very closely. In contrast, open-source models were generally more stable but struggled with nuanced or complex phrasing.
- Impact of Prompt Categories: Certain prompt categories, like structured formatting, chain-of-thought prompts, ambiguity, target audience, roleplay scenarios, and answer handling, showed higher sensitivity, especially for proprietary models. This means that variations within these categories significantly affected model performance.
- Best and Worst Performing Prompts: For open-source models, concise and direct prompts often yielded better results, while complex or overly structured formats (like JSON or YAML) tended to reduce accuracy. Conversely, proprietary models were more robust to complex formatting and often benefited from prompts that allowed for reasoning or included penalties/incentives.
- Linguistic Quality: Open-source models were negatively impacted by poor linguistic formatting (e.g., misspellings, all caps), whereas proprietary models showed greater robustness to such errors.
- Model Size and Sensitivity: Smaller open-source models (e.g., 1B parameters) exhibited greater prompt sensitivity, likely due to limited capacity and weaker context retention. Larger open-source models (8B-38B) showed more stable behavior.
Also Read:
- Unpacking Prompt Sensitivity in Large Language Models
- The Hidden Vulnerability of LLMs: Performance Drops with Reworded Questions
Prompting Principles for Better Evaluation
Based on their extensive analysis, the researchers propose a set of “Prompting Principles” tailored for both proprietary and open-source LMMs. These principles aim to enable more robust and fair model evaluation:
- For open-source models, concise and direct prompts are generally more effective. Overly detailed or complex instructions can be detrimental.
- Proprietary models, due to their advanced instruction-following capabilities, can handle and even benefit from more structured and detailed prompts, including those that encourage reasoning or provide performance-based feedback.
- Both types of models benefit from prompts that emphasize temporal reasoning for video-based tasks.
The study underscores the importance of careful prompt engineering for reliable LMM evaluation. By understanding and accounting for prompt sensitivity, researchers and developers can achieve more consistent and accurate assessments of these powerful multimodal AI systems.
