TLDR: This research demonstrates that Large Language Models (LLMs), particularly GPT-4o with few-shot prompting, can effectively automate thematic analysis of social media data, achieving high accuracy in replicating expert classifications of xylazine-related discussions. This approach offers a scalable solution for qualitative research and can track evolving public health discourse, reducing the need for extensive manual analysis.
The emergence of large language models (LLMs) has opened new avenues for natural language processing in medicine, particularly for qualitative tasks like synthesis and summarization. However, applying LLMs to inductive thematic analysis, a method for identifying and interpreting patterns in data, has been challenging due to the need for deep interpretive and domain-specific expertise. This research explores the feasibility of using LLMs to replicate expert-driven thematic analysis of social media data, specifically focusing on discussions around xylazine wound management.
The study utilized two datasets from Reddit, a popular social network for candid discussions, related to xylazine. Xylazine is an emerging concern in the opioid crisis, often found in illicit drug supplies and associated with necrotizing skin wounds. The datasets were temporally separated (2014-2023 and 2024-March 2025) to assess how well the models could generalize across different periods, anticipating shifts in discussions mirroring the opioid crisis’s evolution.
The researchers modeled the thematic analysis task as a series of binary classifications, meaning for each theme, the LLM would determine if a post was relevant (1) or not (0), rather than trying to classify a post into multiple categories at once. They employed zero-shot, single-shot, and few-shot prompting strategies. Zero-shot means giving the model no examples, single-shot means one example, and few-shot means a few examples to guide its understanding. Performance was measured using accuracy, precision, recall, and F1-score.
Initial testing on a smaller dataset (DS1) showed that few-shot prompting strategies, particularly with GPT-4o, yielded the best results. GPT-4o with two-shot prompting performed exceptionally well on the validation set (DS2), achieving an accuracy of 90.9% and an F1-score of 0.71. For themes that appeared frequently, the distributions identified by the model closely matched those classified by human experts. For instance, the theme “xylazine use” was identified in 13.6% of posts by the model versus 17.8% by experts, and “MOUD use” (Medications for Opioid Use Disorders) was 16.5% versus 17.8%.
The findings suggest that LLM-based approaches, especially when guided by a few examples, can automate thematic analyses. This offers a scalable way to supplement traditional qualitative research, which is often resource-intensive and time-consuming. The study also observed that LLM-generated themes could track shifts in discourse over time. For example, discussions shifted from wound care in earlier data to concerns about drug impurities and fentanyl-xylazine combinations in later data, aligning with epidemiological trends. This capability could be valuable for public health surveillance, providing early warnings for emerging drug threats.
While closed-source models like GPT-4o showed superior performance, the researchers noted that refined prompt engineering could potentially improve open-source models to close the performance gap. The study acknowledges limitations, such as the challenge of fully automated thematic analysis and the potential for keyword-based filtering to miss relevant content. Future work aims to enhance low-shot methods, improve LLM agreement with human experts through feedback, and design frameworks for purely unsupervised categorization to identify new themes without predetermined categories.
Also Read:
- Comparing AI and Human Approaches to Thematic Analysis in Digital Mental Health Research
- AI’s Role in Reshaping Online Discourse: A Look at LLM-Powered Abusive Text Transformation
This research highlights the significant potential of LLMs to streamline qualitative research processes, making large-scale data analysis more efficient and providing valuable insights for public health initiatives. You can find the full research paper for more details. https://arxiv.org/pdf/2507.10803
