TLDR: The paper introduces POT (Prompt-Only OverThinking), a novel black-box attack framework that uses LLM-based iterative optimization to create subtle, natural adversarial prompts. These prompts induce large language models (LLMs) to generate excessively long and computationally inefficient reasoning chains, known as ‘overthinking,’ without needing external data or model access. Experiments show POT significantly increases reasoning token inflation while maintaining answer accuracy and demonstrating strong transferability across various LLMs and reasoning tasks.
Large Language Models (LLMs) have become incredibly powerful, especially with the advent of Chain-of-Thought (CoT) prompting. This technique allows LLMs to break down complex problems into multiple steps, leading to more sophisticated and accurate solutions. However, this enhanced reasoning capability also introduces a new vulnerability: computational inefficiency, often referred to as ‘overthinking’.
Overthinking occurs when an LLM generates unnecessarily verbose reasoning chains, consuming excessive computational resources without a corresponding improvement in performance. Previous attempts to induce overthinking typically required specific conditions, such as access to external knowledge sources for data poisoning, reliance on retrieving poisoned content, or using obvious, templated prompts. These limitations made such attacks less practical in real-world scenarios.
Introducing POT: Prompt-Only OverThinking
To address these challenges, researchers have proposed a novel black-box attack framework called POT (Prompt-Only OverThinking). Unlike its predecessors, POT eliminates the need for external data access or model retrieval. Instead, it uses an LLM-based iterative optimization process to generate covert and semantically natural adversarial prompts.
The core idea behind POT is to subtly manipulate the LLM’s reasoning trajectory through carefully crafted prompts. These prompts are designed to induce the model to generate more reasoning tokens—essentially making it ‘think’ more extensively—while ensuring the final answer remains correct. This means an attacker can force an LLM to use more resources and take longer to respond, without changing the outcome of the task.
How POT Works
The POT framework operates in three main stages:
- Initial Prompt Constructor: An independent LLM generates a diverse set of initial guiding phrases. These phrases are designed to trigger excessive reasoning without being overtly suspicious or unnatural. Examples include phrases like “You are an experienced logician. Try to analyze the problem step by step from multiple perspectives” or “Please thoroughly examine all prior conditions and logical chains relevant to this problem.”
- LLM-Based Optimizer: Another LLM acts as an intelligent search agent, iteratively refining these candidate prompts. It uses a structured ‘meta-prompt’ that includes the attack objective and historical performance feedback to generate new, more effective guiding phrases. A diversity-aware filtering mechanism ensures that the generated prompts are semantically varied, preventing the optimization process from getting stuck in local optima.
- Prompt Assembler: Finally, an LLM integrates the optimized guiding phrases with the actual user query. This ensures the final adversarial prompt is linguistically fluent and stealthy, seamlessly blending with the original question to trigger overthinking in the target LLM.
Key Objectives of the Attack
The attackers using POT have two primary goals:
- Token Inflation: To make the target LLM generate significantly more reasoning tokens compared to a clean query, thereby increasing computational costs and inference latency.
- Answer Consistency: To ensure that the LLM’s final output remains consistent with the correct answer that would be generated from the original, un-attacked input.
Empirical Results and Impact
Extensive experiments were conducted across various mainstream LLMs, including GPT-o1, Claude-Sonnet-3.7, and Gemini-2.5-Pro, and diverse mathematical reasoning datasets like MathQA, AIME 2024, and MATH-500. The results demonstrated that POT consistently achieved superior performance compared to other existing methods.
For instance, on the MathQA dataset, POT achieved an 8.3x increase in Reasoning Token Inflation (RTI) on GPT-o1, 7.8x on Claude-Sonnet-3.7, and 6.1x on Gemini-2.5-Pro. This significantly outpaced other attacks that rely on external content injection. Crucially, POT maintained high answer accuracy (above 90% in most cases) and achieved high attack hit rates (81% to 90%), indicating its effectiveness without compromising the correctness of the LLM’s output.
Furthermore, POT-generated adversarial prompts showed strong transferability across different model architectures, highlighting its robustness and potential threat in black-box and cross-model attack scenarios.
Also Read:
- Unmasking Privacy Vulnerabilities in AI Recommender Systems: New Attacks on LLMs
- JailExpert: A New Framework for Automated LLM Jailbreaking Through Experience
Potential Defense Strategies
The paper also suggests potential defense mechanisms against such semantic-level prompt injection attacks. These include:
- Semantic Caching: Grouping similar prompts based on their meaning to reuse stored responses, reducing redundant processing.
- Difficulty-Aware Budgeting: Setting dynamic limits on reasoning tokens based on the complexity of the question.
- Attention Dampening: Reducing the influence of guiding phrases during inference to prevent them from hijacking the model’s logical flow.
While these defenses show promise, their implementation in closed-source or API-based LLM systems remains a challenge due to the need for access to internal model mechanisms.
In conclusion, POT represents a significant advancement in black-box overthinking attacks, demonstrating how subtle semantic manipulations can lead to substantial computational inefficiencies in LLMs. This research underscores the importance of developing robust defenses against such sophisticated prompt-based vulnerabilities. You can read the full research paper here: POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization.
