TLDR: A new study introduces TRIALQA, a dataset of Reddit posts on colon and prostate cancer, to evaluate how Large Language Models (LLMs) can identify potential clinical trial participants. While LLMs show promise, especially with fine-tuning and in-context learning, they struggle with complex reasoning and often default to ‘Unknown’. Smaller models like Mistral-7B and even a non-LLM baseline (RoBERTa) proved competitive, highlighting the need for improved LLM reasoning for this critical application.
Clinical trials are a cornerstone of medical advancement, but finding eligible participants is a persistent hurdle. Traditional recruitment methods are often slow and limited by geography. This challenge is being addressed by a new approach that taps into the vast amount of health information people share on social media platforms, combined with the advanced text understanding capabilities of large language models (LLMs).
Researchers have explored whether LLM-driven tools can streamline clinical trial recruitment by identifying potential participants through their social media engagement. This innovative study introduces TRIALQA, a unique dataset compiled from Reddit discussions related to colon cancer and prostate cancer. This dataset is meticulously annotated by experienced professionals, indicating whether a social media user meets specific eligibility criteria for real-world clinical trials and their stated reasons for interest in participating.
The study benchmarked seven widely used LLMs, ranging in size from smaller 7-8 billion parameter models to larger models with 70 billion parameters or more. These models were tested using six distinct training and inference strategies, including direct inference, in-context learning (ICL), self-consistency, and chain-of-thought (CoT) reasoning, as well as user-level and entry-level fine-tuning. A non-LLM baseline, RoBERTa, a transformer-based model fine-tuned for natural language inference, was also included for comparison.
Interestingly, for direct prompting on the colon cancer dataset, some smaller LLMs, like Mistral-7B, surprisingly outperformed their larger counterparts. This might be due to larger models sometimes generating overly verbose outputs that don’t strictly adhere to the required format, making answer extraction difficult. The in-context learning method, which provides few-shot examples, generally improved performance by guiding models to follow the desired output format. However, self-consistency methods sometimes led to performance degradation, possibly because the limited output space (True, False, Unknown) restricted reasoning diversity.
Fine-tuning the LLMs consistently improved performance for both eligibility criteria and interest reason prediction tasks, suggesting that social media posts contain valuable signals for these attributes. Entry-level fine-tuning, which treats each post as a single sample, generally yielded better results than user-level fine-tuning, likely due to the larger and more diverse dataset it provides. Notably, the non-LLM RoBERTa model often showed competitive or even superior performance compared to many LLMs, indicating that specialized NLI models can be highly effective and more cost-efficient for this task.
The study also revealed that LLMs performed better on simpler, binary interest reason tasks compared to the more complex, three-class eligibility criteria. Criteria involving explicit textual cues, such as age ranges or recent antibiotic use, were easier for LLMs to predict. However, criteria requiring multi-hop reasoning or implicit information extraction posed greater challenges. A common error observed was LLMs defaulting to the “Unknown” label when they struggled to extract implicit information from user posts. Furthermore, the popular Chain-of-Thought (CoT) strategy, designed to induce step-by-step reasoning, showed more benefits for larger models, suggesting they possess stronger inherent reasoning capabilities.
Common error types included incorrect conclusions drawn from otherwise correct reasoning, entirely incorrect reasoning, and confusion between “No” and “Unknown” labels. These findings highlight that while LLMs show considerable promise, they still face significant challenges in performing the complex, multi-hop reasoning necessary for accurately assessing clinical trial eligibility criteria from social media data. The research lays a foundation for future work in automating and improving clinical trial recruitment processes.
Also Read:
- Automating Scientific Benchmarks: How Reasoning Traces Elevate Small Language Models
- Brain Tumor Imaging: Traditional Neural Networks Outperform Large Language Models
For more in-depth information, you can read the full research paper here.
