Designing AI for Impact: A Multi-Tiered Approach for LLM Tools in Resource-Constrained Hospitals

TLDR: A research paper introduces a novel multi-tiered co-design framework for developing large language model (LLM) applications in resource-constrained environments like safety-net hospitals. This framework addresses the challenge of limited access to domain experts and lack of AI expertise by assembling cross-functional teams, decomposing complex tasks into manageable attributes, and using a cascading refinement and validation process, ultimately leading to highly accurate and useful LLM tools for social workers.

Large Language Models (LLMs) hold immense promise for transforming labor-intensive tasks, especially in critical, resource-constrained environments like safety-net hospitals. These hospitals provide essential medical and social services to underserved communities but often grapple with severe staff and resource shortages. While LLMs could significantly amplify their services, developing truly useful AI applications for such settings presents a unique challenge: a significant disconnect often exists between LLM developers, who may lack specific domain expertise (like clinical social work), and domain experts, who have deep knowledge but are severely time-constrained.

The Challenge of Co-Design in Resource-Limited Settings

Traditional approaches to co-designing LLM applications typically assume frequent, iterative communication between developers and domain experts. However, this model breaks down in environments where experts, such as social workers (SWs) in a safety-net hospital, are overwhelmed with large caseloads and can only spare minimal time. This paper explores this challenge through a real-world case study: building an LLM application to summarize patients’ social needs for inpatient social workers.

The primary responsibility of these social workers is to address social and behavioral determinants of health (SBDH) and facilitate safe patient discharge. A crucial, time-consuming step is “pre-chart summarization,” where SWs manually review fragmented patient records to identify SBDH across various domains like housing, food, and transportation. This task alone consumes over 300 hours per week for a team of 23 SWs, highlighting a critical bottleneck that LLMs could alleviate.

The research identified four key obstacles:

Resource Constraints and Expertise Gap: Data scientists (DSs) lacked social work expertise, while SWs had very limited availability.
Lack of Gold-Standard Summaries: Pre-chart summarization was an undocumented, tacit process, meaning no existing examples or templates for training LLMs.
Underspecified Design Requirements: It was unclear what specific information from patient records was most relevant for a useful summary.
Evaluation Challenges: Rigorous validation of LLM extractions in a high-stakes domain like healthcare required balancing accuracy with resource use.

Initial attempts using existing LLM design techniques failed. These failures underscored two critical learnings: first, the need to “bootstrap” the development process without heavy initial SW input, and second, the necessity of breaking down complex, free-form summaries into individual, manageable attributes.

A Novel Multi-Tiered Co-Design Framework

To overcome these barriers, the research introduces a modular, multi-tiered co-design framework designed for resource-constrained settings. This framework involves four key steps:

Step 1: Assemble a Team that Closes the Gap

Beyond just data scientists and domain experts, the framework advocates for including Cross-Functional Experts (CFEs). These individuals possess moderate availability and interdisciplinary expertise, bridging the knowledge and availability gap between DSs and SWs. In this study, CFEs were clinical care providers and researchers with some data science exposure, ensuring that SWs’ valuable time was consulted only when absolutely necessary.

Step 2: Decompose the Free-Form Summary into Attributes

Instead of aiming for a single, free-form summary, the task was broken down into 15 specific, semi-structured attributes based on established SBDH screening areas and hospital needs (e.g., housing stability, food security, mental health). This modularity allowed for independent tuning, optimization, and validation of each attribute, making the task more manageable and the LLM extractions more accurate. It also enabled a modular user interface, where some elements could be human-validated while others highlighted the need for live verification by SWs.

Step 3: Tiered Refinement of Summary Attributes

This step involved an iterative, three-tier cascading approach to refine attribute definitions and prompts:

Tier 1 (Data Scientists): DSs initiated the process by bootstrapping attribute definitions and prompts using readily available organizational artifacts, such as SW onboarding materials and past SW notes (for in-context learning examples).
Tier 2 (Cross-Functional Experts): CFEs annotated a small set of notes, providing concrete feedback that helped DSs improve the clarity and specificity of most attributes.
Tier 3 (Social Workers): SWs were consulted only for high-value cases or attributes where confusion persisted. This targeted approach maximized the efficiency of their limited time, leading to crucial refinements, such as broadening “interpersonal violence” to “safety concerns” for better accuracy and validation feasibility.

Step 4: Tiered Evaluation of LLM Extractions

Validation also followed a multi-tiered approach:

Tier 1 (LLM-as-a-judge): An LLM was used to judge the LLM application’s extractions against the original patient notes. This low-cost method provided an approximate measure of prompt clarity and progress across iterations, showing how refinements incrementally increased concordance.
Tier 2 (Human Validation): A carefully designed human annotation study involving CFEs and DSs was conducted on a statistically determined sample of notes. This minimized human annotation costs while ensuring rigor.

Also Read:

Results and Impact

Experiments using a PHI-compliant version of GPT-4o on clinical notes demonstrated the effectiveness of this framework. The tiered prompt tuning significantly improved concordance with the LLM-as-a-judge, showing an 18% overall improvement in prompt clarity from the initial to the final version. More importantly, the final LLM application achieved a high overall agreement rate of 0.89 with human annotators, matching the agreement rate among human annotators themselves. While the LLM-as-a-judge proved useful as a general guide, human validation remained critical for comprehensive quality control, especially for attributes like “Patient Contacts” and “Reason for Consult” where LLM-judge scores were high but human agreement was lower.

This research highlights that new co-design strategies are essential for developing LLM applications in low-resource settings. The modular, multi-tier cascading framework facilitates efficient, collaborative development, bridging the critical gap between AI developers and domain experts. The full research paper can be found here: Research Paper.