TLDR: This research explores using Large Language Models (LLMs) for generating and summarizing cloud access control policies. While LLMs can create syntactically correct policies, they often have semantic errors, leading to overly permissive or restrictive rules. The paper introduces “PolicySummarizer,” a novel approach that combines LLMs with formal methods (automata and model counting) to accurately summarize existing policies and their differences in human-readable regular expressions, significantly outperforming traditional methods in precision and interpretability.
Cloud computing has become an indispensable part of our digital world, hosting an ever-increasing number of services and vast amounts of private data. Protecting this data is paramount, and administrators rely on access control policies to define who can access what. However, these policies are typically written manually, a process that is not only complex but also prone to errors. A single mistake in an access control policy can lead to severe data breaches, as evidenced by past incidents where millions of customer records were exposed.
Given the success of Large Language Models (LLMs) in automating tasks like code synthesis and summarization, there’s a natural inclination to explore their potential in generating and understanding access control policies. This paper, “Exploring Large Language Models for Access Control Policy Synthesis and Summarization,” by Adarsh Vatsa, Bethel Hall, and William Eiers from Stevens Institute of Technology, delves into this very topic, investigating how LLMs can be leveraged to improve cloud security.
The Challenge of Policy Synthesis with LLMs
The research highlights a significant hurdle: while LLMs can generate policies that look syntactically correct, they often struggle with semantic accuracy. This means the policies might not behave exactly as intended, leading to what the authors call “permissiveness issues.” For instance, non-reasoning LLMs generated semantically equivalent policies only 45.8% of the time, while even more advanced reasoning LLMs achieved 93.7%. This gap is critical in security, where even subtle errors can create major vulnerabilities.
The paper identifies a “Verifiable Synthesis Paradox”: if a user provides a perfectly clear, formal specification, an LLM isn’t really needed, as existing tools can compile it directly with formal guarantees. But if the requirements are in natural language, which is often ambiguous, the LLM has to interpret these ambiguities. The problem is, there’s no “ground truth” to verify the LLM’s interpretation before the policy is deployed, making it hard to guarantee correctness.
PolicySummarizer: A Novel Approach to Policy Analysis
To address these challenges, the researchers introduce a novel semantic-based request summarization approach called PolicySummarizer. This tool aims to help administrators understand existing policies more precisely and analyze the subtle deviations in LLM-generated ones. PolicySummarizer combines several advanced techniques:
- Policy Encoding: It translates complex access control policies (like those used in Amazon Web Services) into mathematical formulas.
- Automata Construction: These formulas are then used to build a “deterministic finite-state automaton” (DFA), which essentially maps out all the requests allowed by the policy.
- Regular Expression Extraction: A regular expression (RDFA) is extracted from the DFA. While precise, these extracted expressions are often too complex for humans to easily understand.
- LLM-Powered Simplification: This is where LLMs come in. PolicySummarizer samples a set of allowed requests from the complex RDFA and feeds these samples to an LLM (specifically, Claude-4-Sonnet, chosen for its regex generation capabilities). The LLM then generates a simplified, more human-readable regular expression (RLLM).
- Quantitative Verification: Crucially, PolicySummarizer doesn’t just trust the LLM. It uses “model counting techniques” to quantify the similarity between the original, precise RDFA and the LLM-simplified RLLM. This ensures that the simplified version accurately reflects the policy’s true behavior. If the similarity isn’t high enough, the more precise (though complex) RDFA is returned.
This approach can also be used to highlight the semantic differences between two policies, providing clear summaries of what one policy allows that the other doesn’t, and vice-versa. This is particularly useful for identifying subtle errors in LLM-synthesized policies compared to their intended originals.
Also Read:
- AI-Powered Design for Cloud Systems: LLMs and Simulators Optimize Distributed Architectures
- Simplifying AI Planning: How Large Language Models Create Abstract Domains
Key Findings and Impact
The experimental evaluation of PolicySummarizer yielded promising results:
- It successfully generated precise resource characterizations for policies, producing simplified regular expressions that were significantly shorter than the original extracted ones, while maintaining a high average similarity score of 0.89.
- The quality of the summarization depended on the number of sampled strings, with an optimal “sweet spot” found around 1000 samples.
- PolicySummarizer dramatically outperformed a baseline SMT-based approach, achieving a 2.7x improvement in semantic accuracy for regex generalization.
These findings suggest that while LLMs alone might struggle with the precision required for policy synthesis, their combination with formal methods, as demonstrated by PolicySummarizer, offers a powerful tool for policy analysis. It provides human-readable summaries that existing binary verification tools cannot, enabling a “human-in-the-loop” verification process where administrators can easily review and understand complex policy behaviors.
The researchers also note that this technique can be extended to summarize other components of access requests, such as allowed actions or principals, by simply adjusting which elements are projected from the multi-track DFAs. This work paves the way for improving policy generation and analysis using quantitative analysis techniques in conjunction with LLMs.
For more in-depth technical details, you can read the full research paper here: Exploring Large Language Models for Access Control Policy Synthesis and Summarization.
