TLDR: “Plan-and-Write” is a prompt engineering method that enables precise length control in Large Language Models (LLMs) without retraining. It uses a two-phase approach (planning with explicit word counting and verification) to guide LLMs to adhere to specific length constraints. Experiments show it significantly improves length fidelity and maintains or enhances output quality for most models, offering a practical solution for production environments.
Large Language Models (LLMs) have become incredibly powerful, but controlling the exact length of their responses remains a significant challenge. Whether it’s a voice assistant needing a concise answer or a research summary requiring comprehensive detail, precise length control is crucial for many real-world applications. Current methods often involve expensive model retraining or complex tools, making them impractical for many businesses.
Introducing Plan-and-Write: A Smart Prompt Engineering Solution
A new research paper, “Plan-and-Write: Structure-Guided Length Control for LLMs without Model Retraining,” introduces an innovative prompt engineering methodology that tackles this problem head-on. Authored by Adewale Akinfaderin, Shreyas Subramanian, and Akarsha Sehwag from Amazon Web Services, this approach allows for precise length control without the need for costly model retraining or complex inference-time tooling. This makes it an immediately deployable solution for production environments.
The core idea behind Plan-and-Write is to guide the LLM through a deliberate planning and word-counting process directly within the prompt. Instead of simply telling the model to summarize in ‘X’ words, the method encourages the model to actively track and adhere to specified length constraints. This structure-guided approach leverages the LLM’s inherent capabilities for self-monitoring and metacognition.
How Plan-and-Write Works
The methodology breaks down the generation process into two distinct phases:
- Planning Phase: In this initial step, the model explicitly counts words as it drafts content. This creates “length awareness,” transforming the abstract length constraint into a concrete, procedural task with clear progress indicators.
- Verification Phase: After drafting, the model reconstructs the content into a coherent, final output that precisely matches the specified length. This separation allows the model to first focus on meeting the word count accurately and then on ensuring the text is fluent and well-structured.
This two-phase structure reduces the cognitive burden on the model, allowing it to address one constraint at a time. It doesn’t require new model capabilities but rather helps the LLM apply its existing skills more effectively, such as counting and following multi-step instructions.
Experimental Validation and Key Findings
The researchers conducted extensive evaluations across six state-of-the-art LLMs, including Claude 3 Haiku, Claude 3.5 Sonnet, Llama 3.1 70B, and Mistral Large. The experiments focused on document summarization tasks with various target lengths, ranging from 20 to 5000 words. The primary metric for success was Mean Absolute Percentage Deviation (MAPD), which measures the percentage error between the generated and target word counts.
The results were compelling: the Plan-and-Write approaches (dubbed “Thinking V1” and “Thinking V2” in the paper) significantly improved length fidelity for four out of six models compared to standard prompting techniques. For instance, Claude 3.7 Sonnet showed a remarkable 37.6% improvement in length adherence. While some models, like Llama 3.1 70B, already exhibited strong length control with standard prompts, the structure-guided method proved particularly beneficial for mid-tier models.
Interestingly, the study also found that most models performed better with longer target lengths (500+ words) and that the Plan-and-Write method consistently reduced the variance in output length, leading to more predictable results.
Maintaining Quality While Controlling Length
A crucial aspect of this research was ensuring that improved length control did not compromise the quality of the generated content. Using an “LLM-as-a-Judge” evaluation method, the researchers assessed output quality across four dimensions: correctness, completeness, faithfulness, and relevance. The Plan-and-Write approach not only improved length fidelity but also maintained or even enhanced summary quality, achieving higher scores in faithfulness, completeness, and relevance compared to vanilla prompting.
Also Read:
- Adaptive Effort Control: AI Models Learn to Optimize Reasoning for Cost and Accuracy
- Balancing Act: How Efficient Fine-Tuning Shapes LLM Safety and Fairness
Considerations and Future Directions
While highly effective, the Plan-and-Write methodology does have some limitations. Its benefits vary across models, and its effectiveness can decrease for very long target lengths (beyond 500 words). There’s also a computational trade-off: thinking prompts require slightly more tokens and longer inference times compared to simpler prompts, though this might be acceptable given the improved control. The quality evaluation, relying on LLM-as-a-Judge, also carries inherent biases.
Despite these points, Plan-and-Write offers a practical and immediately deployable solution for developers needing precise length control in LLMs without the overhead of model retraining. This research opens doors for future exploration into combining prompt engineering with lightweight inference modifications and developing adaptive prompting strategies. You can read the full research paper here.
