TLDR: Researchers have developed an AI agent that leverages property-based testing (PBT) to autonomously identify bugs across the Python ecosystem. This agent analyzes code, infers properties, generates and executes PBTs, and creates detailed bug reports. An extensive evaluation across 100 popular Python packages revealed that 56% of the generated bug reports were valid, including critical issues in libraries like NumPy. This approach demonstrates a scalable and rigorous method for automated software auditing, with several discovered bugs leading to successful patch merges.
Software testing is a critical part of development, ensuring that programs work as intended. Traditionally, this involves example-based testing, where developers write specific inputs and check for expected outputs. While effective for known scenarios, this approach can miss subtle bugs, especially in complex codebases, as it relies on the developer to anticipate every edge case.
A more robust method, Property-Based Testing (PBT), aims to verify general properties or invariants that should hold true for all inputs within a defined domain. Instead of specific examples, PBT frameworks automatically generate a diverse set of inputs and check if the defined properties consistently hold. If a property fails, the framework identifies a counterexample, revealing a potential bug.
Introducing Agentic Property-Based Testing
Despite its theoretical advantages, PBT has seen limited adoption due to the challenge of identifying meaningful properties to test, which often requires significant domain expertise and time. This is where recent advancements in Large Language Models (LLMs) come into play. Researchers have developed an innovative approach called “agentic property-based testing,” which leverages the multi-step reasoning capabilities of LLM-based agents to overcome these challenges.
This new agent is designed to autonomously analyze Python modules, infer function-specific and cross-function properties from code and documentation, synthesize and execute PBTs, reflect on test outputs to confirm genuine bugs, and finally, generate actionable bug reports for developers. The agent is built upon Anthropic’s Claude Code, a terminal-based coding agent, and is implemented as a natural language prompt, making it adaptable to other agent implementations.
How the Agent Works: A Six-Step Cycle
The agent follows a systematic six-step process:
1. Analyze the Target: It first determines whether it’s testing a module, a specific file, or a function.
2. Understand the Target: The agent uses Python introspection to read documentation, function signatures, and source code, understanding the module’s structure and how functions are used.
3. Propose Properties: Based on its understanding, it looks for high-value property patterns such as invariants (e.g., the length of a filtered list is less than or equal to the original), round-trip properties (e.g., encoding and then decoding data should return the original data), inverse operations, mathematical properties, and metamorphic properties. The agent prioritizes properties strongly supported by existing code or documentation.
4. Write Tests: It translates these proposed properties into Hypothesis property-based tests, intelligently constraining inputs to the relevant domain.
5. Execute and Triage Tests: The tests are run using `pytest`. For any failures, the agent applies a bug triage rubric to check for reproducibility, legitimacy (realistic usage, claimed property), and impact. If it’s a false alarm, the agent refines its testing strategy; if it’s a genuine bug, it proceeds to reporting.
6. Report Bugs: If convinced a bug is genuine and reproducible, it creates a standardized bug report in Markdown format, including a summary, the failing PBT, a reproduction script, an explanation of why it’s a bug, and potentially a proposed fix.
Extensive Evaluation and Real-World Impact
The effectiveness of this agentic PBT approach was demonstrated through a large-scale evaluation across 100 popular Python packages, encompassing 933 modules. These packages included hand-selected standard library modules (like `json`, `pathlib`), popular third-party PyPI packages (like `numpy`, `pandas`, `requests`), and a random sample from the top 5,000 PyPI packages.
The agent generated 984 bug reports, identifying issues in 84.2% of the modules tested. After manual review, a significant portion of these reports were confirmed as valid bugs (56%), with 32% deemed valid and worth reporting to maintainers. For the highest-scoring bugs, the validity rate was even higher, at 86%.
The bugs discovered spanned a diverse range of failure modes, from serialization issues and numerical precision errors to flawed cache implementations. Notably, the agent autonomously found a bug in NumPy, a foundational numerical library, where the `numpy.random.wald` function sometimes produced negative samples despite the Wald distribution being mathematically defined for positive values only. This bug was acknowledged by maintainers, and a patch was successfully merged.
Other significant discoveries included a logic error in `aws-lambda-powertools` where `slice_dictionary` returned duplicate chunks, a critical hashing bug in `cloudformation-cli-java-plugin` where all list inputs hashed to the same value, and a malformed HSL color string in `tokenizers` due to a missing parenthesis. In total, 3 patches were successfully merged into popular libraries.
The evaluation also provided insights into the cost-effectiveness, with an estimated cost of $9.93 per valid bug found, suggesting a scalable method for software auditing. The researchers have made their code and artifacts available for public access at: https://github.com/mmaaz-git/agentic-pbt.
Also Read:
- Advancing Software Testing with AI Agents and Hybrid Knowledge Systems
- SLEAN: A Simple Approach to Coordinating Multiple AI Models for Reliable Software Debugging
Conclusion
This research highlights that combining LLMs with property-based testing offers a rigorous and scalable method for autonomously testing software, systematically uncovering bugs that traditional testing might miss. While the agent has limitations, such as distinguishing intentional design violations from bugs, its ability to find genuine, diverse issues in widely-used libraries establishes a new paradigm for automated software auditing. As LLMs continue to improve, agentic PBT could become an increasingly valuable complement to traditional testing, helping developers explore code behavior and find bugs more effectively.
