TLDR: PACO (Adaptive Planning for Multi-Attribute Controllable Summarization) is a new training-free framework that uses Monte Carlo Tree Search to enable large language models to create summaries that precisely meet multiple user-specified attributes (like length, topic, or speaker). It works by progressively adjusting attributes through a planning process, outperforming existing LLM-based and fine-tuned methods while maintaining summary quality, even with smaller models.
Controllable summarization is a crucial area in artificial intelligence, aiming to produce summaries tailored to specific user needs, such as desired length, topic, or level of detail. However, achieving this with large language models (LLMs) has been challenging because different attributes often interact in complex ways, making it difficult for models to satisfy multiple constraints simultaneously. Traditional approaches often require extensive fine-tuning for each specific attribute, which limits their flexibility and ability to adapt to new preferences.
A new framework, called Adaptive Planning for Multi-Attribute Controllable Summarization (PACO), addresses these challenges by reframing the summarization task as a planning problem. Instead of trying to control all attributes at once, PACO uses a customized Monte Carlo Tree Search (MCTS) algorithm to progressively refine summaries. This training-free approach allows the model to adaptively discover the best order of sequential attribute adjustments, leading to summaries that effectively meet all specified constraints.
How PACO Works
PACO integrates the MCTS algorithm into the multi-attribute controllable summarization process. In this framework, each ‘node’ in the search tree represents a complete summary, and each ‘action’ corresponds to adjusting a single attribute (e.g., extractiveness, length, specificity, topic, speaker). This summary-level node definition helps reduce the complexity of the search space compared to token or sentence-level approaches.
The process begins with an initial summary that attempts to control all attributes. From this root node, PACO explores different control orders. It uses a selection process based on the Predictor Upper Confidence Tree (PUCT) algorithm to choose promising paths. When a leaf node is reached, it’s expanded by generating child nodes for all possible attribute adjustments. The evaluation of each node involves a ‘local reward’ that measures immediate improvements and a ‘heuristic score’ that assesses the likelihood of meeting all remaining attributes. Finally, a ‘backpropagation’ step updates the values and visit counts along the search path. The ultimate decision involves selecting the summary with the highest ‘degree’ of attribute control from the entire tree, allowing for flexible control over a subset of attributes rather than strictly enforcing all of them.
Attributes are categorized into two types: ‘deterministic’ attributes, like extractiveness, length, and specificity, which must match exact user targets; and ‘non-deterministic’ attributes, such as topic and speaker, where higher values indicating better alignment are preferred.
Impressive Performance
Extensive experiments across various domains and models demonstrate PACO’s robust multi-attribute controllability. It was tested on datasets like MACSumDoc, MACSumDial, and DialogSum, using different LLMs including Llama-3.2-1B-Instruct, Llama-3.3-70B-Instruct, and Qwen2.5-7B-Instruct. PACO consistently outperformed both LLM-based self-planning models and fine-tuned baselines.
Remarkably, PACO, even when paired with a smaller Llama-3.2-1B model, achieved controllability comparable to that of the much larger Llama-3.3-70B baselines. With larger models, PACO delivered superior control performance, surpassing all competitors. Crucially, PACO achieves these gains without sacrificing the overall quality of the summaries, as it incrementally adjusts attributes rather than forcing all constraints at once, which can sometimes compromise quality.
In contrast, LLM-based self-planning methods, both implicit and explicit, struggled to generate effective plans, often performing worse than the baseline models. This highlights the need for a more structured planning strategy like PACO to guide the generation process effectively.
Also Read:
- DeepSearch: Enhancing Language Model Reasoning Through Integrated Tree Search Training
- Adaptive Learning: How On-Demand Expert Help Boosts AI Reasoning
Future Directions
While PACO offers strong controllability, the tree search can be computationally expensive, especially for deeper simulations. Future work aims to explore search-time heuristics to reduce this cost. Additionally, expanding PACO to optimize for broader quality dimensions beyond attribute alignment, such as coherence, consistency, relevance, and fluency, could lead to even more comprehensive and user-tailored summarization. You can read the full research paper here: Adaptive Planning for Multi-Attribute Controllable Summarization with Monte Carlo Tree Search.
