TLDR: A study interviewed 24 knowledge workers in high-risk fields (law, healthcare, finance) to understand their concerns and strategies for using Large Language Models (LLMs) safely. Professionals are worried about data leakage, intellectual property infringement, and inaccurate outputs. They currently use manual methods like input sanitization and extensive verification, but these are limited by a lack of systemic support and clear guidance. The research proposes designing human-centered, compliance-driven NLP tools that offer proactive risk alerts, support professional judgment, and provide automated compliance scaffolding to bridge the gap between AI capabilities and real-world compliance needs.
Large Language Models (LLMs) are rapidly transforming how professionals work across various industries, from law and healthcare to finance. While these AI tools offer significant productivity boosts, their widespread adoption in high-risk domains introduces complex and often underexplored compliance challenges. A recent study, titled Towards Human-Centered RegTech: Unpacking Professionals’ Strategies and Needs for Using LLMs Safely, delves into how highly-skilled knowledge workers perceive and navigate these risks, highlighting a critical gap between current AI capabilities and real-world compliance needs.
Authored by Siying Hu, Yaxing Yao, and Zhicong Lu from City University of Hong Kong, Johns Hopkins University, and George Mason University respectively, the research involved semi-structured interviews with 24 experts. These professionals, with at least three years of experience, were drawn from eight knowledge-intensive industries, with a particular focus on high-compliance-pressure fields like law and healthcare. The goal was to understand their specific concerns, the strategies they spontaneously employ to mitigate risks, and the challenges they face in doing so.
Key Compliance Risks Identified
The study revealed that professionals are primarily concerned about three major compliance risks when using LLMs:
- Sensitive Information Leakage: Experts worry that confidential data inputted into LLMs could be memorized by the model, integrated into future versions, or exposed through sophisticated attacks. This poses direct liabilities under data protection regulations such as GDPR and CCPA, turning the AI tool into a potential source of unacceptable risk.
- Intellectual Property (IP) Infringement: A significant concern is the lack of provenance and attribution for LLM outputs. Professionals fear that model-generated content might infringe on existing copyrighted material, or that their unique prompts and proprietary information could be absorbed into the model’s training data without proper attribution.
- Output Inaccuracy (Hallucinations): The risk of LLMs generating factually incorrect or misleading information, often referred to as “hallucinations,” is a direct threat to professional liability. In fields where accuracy is paramount, such errors can have severe legal, ethical, and reputational repercussions.
Professionals’ Spontaneous Mitigation Strategies
In the absence of robust built-in technical safeguards, knowledge workers have developed their own manual, user-side control frameworks:
- Proactive Input Sanitization: This involves manually redacting sensitive entities and reducing the specificity of prompts. By limiting the detail in their inputs, professionals aim to minimize the model’s ability to infer or retain confidential information.
- Intensified Human-in-the-Loop Verification: Adopting a “zero-trust” policy, experts treat all LLM outputs as unverified drafts. This necessitates a thorough post-processing validation layer, where professionals manually audit and verify the content. While this adds a crucial layer of safety, it often negates the efficiency gains that LLMs promise, essentially adding to their workload rather than reducing it.
Challenges and the Need for Systemic Support
Despite these diligent efforts, the study found that user-driven mitigation strategies are often insufficient due to systemic challenges within the current AI ecosystem. A core issue is the lack of model explainability and causal traceability, making it impossible to conduct formal risk assessments or audits. This technical opacity creates a regulatory vacuum, leaving professionals operating in a “gray area” without clear AI use policies or a defined chain of responsibility among developers, providers, deployers, and users.
Also Read:
- European AI Developers Grapple with Privacy: A Look at Risks and Real-World Solutions
- Unpacking How Large Language Models Perceive and Manage Risk
Designing Human-Centered, Compliance-Driven NLP Tools
The research offers critical design implications for the future of Human-Centered NLP and HCI, advocating for a new generation of RegTech systems that proactively support expert compliance workflows:
- Transparency by Design: Future NLP systems should act as proactive guides, offering real-time, context-aware compliance risk alerts tailored to the user’s industry and specific task. For example, a legal professional inputting case information could receive automatic highlights of potential client privacy risks.
- Interaction Design Supporting Professional Judgment: Systems should embed values like fairness, privacy, and accountability. This means providing clear feedback channels and granting users final review and modification rights over AI suggestions, thereby empowering rather than replacing expert judgment.
- Practical NLP Tools and Scaffolding: To reduce the compliance burden, future systems need concrete automation tools. This includes NLP modules that can automatically identify and de-sensitize personally identifiable information (PII) or trade secrets during input. Additionally, scaffolding-style interactive workflows could guide users through high-risk tasks with compliance checklists and standard operating procedures.
In conclusion, this study underscores the urgent need for a human-centered approach to developing LLM-based RegTech. By understanding the real-world dilemmas and practical wisdom of frontline experts, we can build trustworthy, compliance-aware NLP tools that genuinely support professionals in safely leveraging the power of AI.
