TLDR: A new research paper introduces a method for automatically generating high-quality, task-specific prompts for large language models (LLMs). The system builds a knowledge base that maps task clusters to effective prompting techniques. When a user describes a task, the system identifies the most relevant cluster and dynamically combines techniques to create a tailored prompt. Experiments on the BIG-Bench Extra Hard dataset show superior performance compared to standard prompts and existing automatic prompt generators, with further gains achieved through temperature optimization. This framework aims to make LLM utilization more accessible to non-experts.
In the rapidly evolving landscape of large language models (LLMs), crafting effective prompts is a critical skill, often referred to as prompt engineering. However, this process typically demands specialized knowledge and a deep understanding of the specific task at hand, creating a significant barrier for many users. A new research paper, “Automatic Prompt Generation via Adaptive Selection of Prompting Techniques,” introduces a novel solution to this challenge.
Authored by Yohei Ikenoue, Hitomi Tashiro, and Shigeru Kuroyanagi from Spike Studio Inc., this paper proposes a method that automatically generates high-quality, task-appropriate prompts without relying on pre-existing templates. Instead, it adaptively selects prompting techniques based on a user’s abstract description of their task. This innovation aims to democratize access to LLM capabilities, allowing non-experts to leverage these powerful AI tools effectively.
The Core Idea: A Knowledge Base of Prompting Techniques
The heart of the proposed system is a sophisticated knowledge base. This base is constructed by first clustering diverse tasks based on their semantic similarity. Imagine grouping tasks like “summarizing a document” and “extracting key information” into a ‘text comprehension’ cluster. Once these task clusters are defined, the system associates each cluster with a set of effective prompting techniques.
When a user provides a description of their task, the system analyzes it, assigns it to the most relevant task cluster, and then dynamically generates a prompt by integrating techniques drawn from its knowledge base. This adaptive approach ensures that the generated prompt is tailored to the specific needs of the user’s task, rather than being a generic, one-size-fits-all solution.
How It Works: Two Key Phases
The system operates in two main phases: knowledge base construction and prompt generation.
In the **knowledge base construction phase**, the researchers define task clusters by vectorizing task descriptions and names using an embedding model (gemini-embedding-exp-03-07). An LLM (gemini-2.5-pro) then helps assign semantic descriptions to these clusters. Crucially, effective prompting techniques are mapped to each cluster. The paper identifies about 15 versatile techniques, categorized into Role Assignment (e.g., Role Playing), Emotional Stimulus (e.g., Emotion Prompting, Stress Prompting), Reasoning (e.g., Chain-of-Thought, Least-to-Most Prompting, Plan-and-Solve Prompting), and Others (e.g., Decomposed Prompting, Scratchpad Prompting). For each cluster, the system selects a combination of three to four techniques based on predefined constraints, ensuring a multifaceted approach to prompt design.
The **prompt generation phase** begins when a user inputs a task description. This description is vectorized and matched to the most semantically similar task cluster in the knowledge base. Once the cluster is identified, the corresponding prompting techniques are retrieved. Finally, the user’s task description and the selected techniques are fed into an LLM (gemini-2.5-pro) to generate the final, high-quality prompt.
Experimental Validation and Promising Results
To evaluate the method’s effectiveness, experiments were conducted on 23 challenging tasks from the BIG-Bench Extra Hard (BBEH) dataset. The proposed method was compared against standard prompts provided in the original BBEH paper and prompts generated by Anthropic’s Prompt Generator.
The results were compelling. The proposed method achieved an arithmetic mean accuracy of 28.0 across tasks, outperforming the Original prompts (23.9) and Anthropic’s tool (24.7). Even more significantly, its harmonic mean score, which emphasizes performance on lower-scoring (and thus more difficult) tasks, was 12.5, surpassing Original (9.7) and Anthropic (10.5). This indicates a more stable and robust performance, especially on complex problems.
Specific tasks like Object Counting and Spatial Reasoning saw notable improvements, demonstrating the method’s strength in tasks requiring step-by-step numerical processing and multi-stage reasoning. However, tasks requiring intuitive visual understanding, such as Geometric Shapes, sometimes saw a decrease in performance, highlighting areas for future refinement.
Beyond Prompts: The Role of Temperature Optimization
The research also explored the impact of runtime parameters, specifically the ‘temperature’ setting of the LLM, which controls the balance between determinism and creativity in its responses. By optimizing the temperature parameter for individual tasks, the proposed method’s performance further improved, reaching an arithmetic mean of 28.5 and a harmonic mean of 13.3. This suggests that a comprehensive optimization strategy should consider both prompt content and LLM execution parameters.
Also Read:
- Enhancing LLM Performance Through Direction-Guided Prompt Evolution
- Capsule Prompt Tuning: Efficiently Guiding Large Language Models
A Foundation for the Future
This research lays a strong foundation for streamlining and standardizing prompt creation, making LLMs more accessible and effective for a wider audience. While the current knowledge base is tailored to BBEH tasks, the framework itself is highly adaptable and can be customized for various domain-specific applications, such as financial analysis or manufacturing defect detection, with significantly reduced expert design costs. Future work aims to incorporate dynamic updates to the knowledge base based on user feedback and to integrate functionality for LLMs to predict prompt effectiveness. For more details, you can read the full paper here: Automatic Prompt Generation via Adaptive Selection of Prompting Techniques.
