TLDR: PAGE (Prompt Augmentation for text Generation Enhancement) is a framework that improves the quality and control of text generated by large language models (LLMs). It uses simple, specialized auxiliary modules (like classifiers or extractors) to process input text and create an “enriched prompt” with structured information. This enriched prompt then guides the LLM to produce better outputs, reducing the need for extensive LLM fine-tuning or large datasets. A proof-of-concept in software requirements generation showed significant improvements over a baseline, demonstrating PAGE’s effectiveness and adaptability.
Large Language Models (LLMs) have revolutionized text generation, producing fluent and coherent content across various tasks. However, when faced with highly specific tasks or particular output requirements, these powerful models can sometimes fall short. Traditionally, improving their performance in such scenarios often demands extensive fine-tuning, which requires vast amounts of additional data and significant computational resources.
Addressing this challenge, researchers Mauro José Pacchiotti, Luciana Ballejos, and Mariel Ale from Universidad Tecnológica Nacional have introduced PAGE (Prompt Augmentation for text Generation Enhancement). This innovative framework offers a simpler, more modular, and adaptable approach to assist LLMs without the need for complex auxiliary generative models or massive retraining efforts.
What is PAGE and How Does It Work?
PAGE operates on the principle that LLMs can generate superior results if their input prompts are enriched with structured, inferred information derived from the original text. Instead of relying on another large generative model, PAGE employs simple, specialized auxiliary modules to perform these inferences.
The framework is built around three core components:
- Auxiliary Modules: These are lightweight models, such as classifiers, entity extractors, or sentiment analyzers. Their role is to process the initial input text and generate structured metadata or insights. For instance, a classifier might identify the category of a sentence, while an extractor might pull out key entities. Crucially, these modules are simpler to train, requiring less computational power and smaller datasets compared to fine-tuning an entire LLM.
- Prompt Composer: This component acts as an orchestrator. It takes the structured output from one or more auxiliary modules and combines it with the original text. Using configurable templates, it constructs an “enriched prompt” – a more informative and guided input for the generative model.
- Generative Model: This is the LLM itself (e.g., Llama, GPT, T5). With the enriched prompt as its input, the generative model is guided to produce a higher quality, more controlled, and structured output, often without needing any fine-tuning.
The PAGE process is straightforward: A user or system provides a base text. Auxiliary modules process this text to generate structured information. The prompt composer integrates this information with the original text to build an enriched prompt. Finally, the generative model produces the desired output from this enriched input. This modular design allows for easy testing of individual components and adaptation to different domains.
Proof of Concept: Enhancing Software Requirements Generation
To validate their proposal, the researchers conducted a proof-of-concept in the domain of software requirements engineering. The goal was to improve the generation of software requirement expressions using the EARS (Easy Approach to Requirements Syntax) framework, which categorizes requirements into types like Event-driven, Ubiquitous, and State-driven, each with a specific syntactic template.
For this experiment, a small dataset of 253 requirement instances was used. The auxiliary module was a Random Forest classifier, trained to identify the EARS category of a given textual expression. Once classified, the module provided two examples of requirements written in the correct EARS syntax for that specific category. This classifier achieved an accuracy of 82.35% on the test set.
The generative model used was Llama 3.1 (8 billion parameters), deployed locally using Ollama. Three experiments were performed:
- Zero-Shot Baseline: The LLM received only the original text with a basic prompt, without any auxiliary enrichment.
- Dataset-Samples (Ideal Scenario): The LLM received an enriched prompt where the correct EARS category label was directly provided from the dataset, along with relevant examples. This represented the best possible performance.
- PAGE Implementation: The full PAGE framework was used, with the Random Forest classifier predicting the EARS category and providing examples, which were then composed into the prompt for the LLM.
The results, evaluated using ROUGE metrics (ROUGE-1, ROUGE-2, ROUGE-L), demonstrated significant improvements. The PAGE implementation achieved a substantial boost in performance compared to the zero-shot baseline. For instance, ROUGE-2 Recall saw an improvement of over 200%. Remarkably, PAGE’s performance was very close to the ideal scenario, falling only 2 to 4 percentage points below the results obtained when the correct EARS labels were directly provided from the dataset. Qualitative analysis further confirmed that PAGE-generated outputs closely matched the expected EARS syntactic structure, unlike the often-disparate outputs from the baseline.
Also Read:
- A New Approach to Evaluating Reasoning in Large Language Models
- Navigating Document Intelligence with Large Language Models: A Comprehensive Overview
Conclusion and Future Directions
The PAGE framework offers a promising and effective method for enhancing the performance, control, and structural quality of text generated by LLMs. Its simplicity, modularity, and adaptability make it particularly valuable for scenarios with limited computational resources or data for extensive LLM fine-tuning. The success of the conceptual proof in software requirements generation highlights the potential for PAGE across various domains.
The research underscores the critical role of well-designed auxiliary modules; their accurate inferences are key to guiding the generative model effectively. As a future step, the authors propose developing a reusable Python software framework to facilitate the implementation, evaluation, and persistence of PAGE instances, further expanding its practical applications. You can read the full research paper here.
