TLDR: The Prompt Declaration Language (PDL) is a novel declarative approach to prompt engineering for LLMs, designed to simplify complex agentic programming. By bringing prompts to the forefront and abstracting underlying complexities, PDL improves programmer productivity and allows for better customization and optimization. A case study with a CISO Compliance Agent demonstrated that using PDL led to up to a 4x performance improvement, especially with smaller LLMs, primarily by reducing tool call failures through its flexible, two-stage architecture and custom parsing capabilities.
Prompt engineering, the art of crafting effective instructions for large language models (LLMs), has long been a complex challenge. Existing frameworks often either hide this complexity behind rigid interfaces or offer pre-set patterns that are difficult to customize, making it hard to build sophisticated AI agents.
Introducing the Prompt Declaration Language (PDL)
A new approach, the Prompt Declaration Language (PDL), aims to simplify this by bringing prompts to the forefront. PDL is a novel way to represent prompts, allowing for both manual and automatic tuning. It captures how LLM calls are composed with rule-based code and external tools, abstracting away the intricate details. This declarative representation, written in YAML, is designed to boost programmer productivity and is also suitable for optimization.
PDL is a typed language that uses JSON Schema for defining types, ensuring the correct shape of LLM inputs and outputs. It seamlessly integrates with constrained decoding for models that support it and leverages LiteLLM to work with a wide variety of LLMs and providers. This means developers can easily switch between different models without worrying about specific message formatting.
Real-World Application: The Compliance Agent Case Study
The utility of PDL was demonstrated through a real-world case study involving a Chief Information Security Officer (CISO) Compliance Agent. This AI agent automates IT compliance tasks, which traditionally require specialized expertise. The initial version of the CISO Agent used a standard ReAct pattern, a common approach for AI agents, implemented with CrewAI.
However, this traditional ReAct pattern often struggled with smaller, more affordable LLMs. A common issue was that these models would produce syntax errors when trying to output both a natural language thought and a structured tool call (ActionSpec) simultaneously. This led to tool call failures and incorrect tool usage.
PDL’s Solution for Enhanced Performance
To address these problems, the CISO Agent developers devised a PDL-based architecture. This new design splits the ‘Think’ stage of the agent’s process into two: ‘Think1’ for natural language output and ‘Think2’ for structured data output (ActionSpec). This two-stage approach prevents the simultaneous output errors seen with smaller models. Traditional agent frameworks typically lack the customization capabilities that PDL provides for modifying core agent workflows.
Furthermore, PDL’s flexibility allowed for a custom Response Parser to handle model-specific output quirks, ensuring correct interpretation of ActionSpec outputs. This highlights PDL’s ability to go beyond the scope of traditional frameworks.
Significant Performance Improvements
Performance evaluations using ITBench showed consistent improvements across all models when using the PDL-based CISO Agent. Notably, smaller models like granite3.2-8b achieved up to a 4x performance improvement. This significant gain primarily stems from a drastic reduction in tool call failures. For instance, with gpt4o-2024-11-20, cases where the tool was not called dropped from 22.4% to 2.4% in the PDL-based implementation. For granite3.2-8b-instruct, these cases decreased from 53.5% to 35.4%, directly contributing to the 4x success rate improvement.
These results underscore how PDL’s fine-grained control over LLM interactions is crucial for optimizing AI agent performance, especially when working with resource-constrained language models. For more details, you can refer to the full research paper: Representing Prompting Patterns with PDL: Compliance Agent Case Study.
Also Read:
- SI-Agent: Automating Clear and Effective Instructions for AI Models
- DSPy Framework Elevates LLM Performance Through Programmatic Prompt Optimization
PDL in the Landscape of Prompting Languages
While other domain-specific languages for prompting exist, such as LMQL or DSPy, they are often embedded in Python. PDL, in contrast, is embedded in YAML, making it easier to manipulate programmatically. Unlike some prompt optimizers that generate unreadable representations, PDL keeps prompts visible and unified with declarative agentic logic, making them easier for developers to understand and tweak. This contrasts with frameworks like AutoGen and CrewAI, where prompts are often buried within the implementation.
In conclusion, PDL offers a powerful and flexible way to represent and manage LLM prompts and agent workflows, leading to improved performance and developer productivity, particularly with the growing use of smaller, more efficient language models.
