TLDR: AutoPK is a new two-stage AI framework that uses large language models (LLMs) and a hybrid similarity metric to accurately extract pharmacokinetic (PK) data from complex scientific tables. It significantly improves precision and recall over direct LLM use, reduces hallucinations, and enables smaller open-source LLMs to outperform commercial systems like GPT-4o Mini, making PK data extraction more scalable and reliable for drug development and public health.
Pharmacokinetics (PK) is a vital field in drug development and regulatory decisions for both human and animal medicine. It directly impacts public health by assessing drug safety and effectiveness. However, extracting PK data from scientific documents has always been a significant challenge. This is because PK data is often buried in complex tables with varied structures and inconsistent terms, making automated retrieval and standardization difficult.
A new framework called AutoPK has been developed to tackle this problem. AutoPK is a novel two-stage system designed for accurate and scalable extraction of PK data from these challenging scientific tables. The core idea is to leverage the power of large language models (LLMs) combined with a smart way to measure similarity between different data points.
How AutoPK Works: A Two-Stage Approach
The AutoPK framework operates in two main stages, each addressing a specific aspect of the data extraction challenge.
The first stage focuses on identifying and extracting the many different ways PK parameters can be expressed. For example, “half-life” might appear as “T1/2,” “HLgamma,” or “Elimination Half-Life.” AutoPK uses large language models to find these variations, along with a “hybrid similarity metric.” This metric combines three ways of comparing text: cosine similarity (for semantic meaning), Levenshtein similarity (for spelling and lexical closeness), and token overlap (for shared words or parts of words). After identifying potential variants, an LLM-based validation step helps to filter out incorrect matches, ensuring high accuracy.
The second stage takes the identified PK parameters and reconstructs them into a clean, standardized, and machine-readable table. First, it filters out irrelevant rows from the original complex table, keeping only the information directly related to the PK parameters. If a parameter is in a column header, the table is temporarily rearranged to ensure consistent processing. Then, the filtered table is converted into a simple key-value text format. For instance, a cell might become something like “50gr@Dosage^Chicken.” This simplified text, along with any table captions or footnotes, is then fed into an LLM. The LLM uses this information to build a new, standardized table in a CSV format, following a predefined structure. Finally, a post-processing step ensures all outputs are consistent, removing empty values, standardizing text to lowercase, expanding abbreviations, and uniformly formatting numbers.
Impressive Performance and Impact
AutoPK was rigorously tested on a real-world dataset of 605 annotated PK tables, including their captions and footnotes. The results showed significant improvements in precision and recall compared to directly using LLMs without the AutoPK framework.
For example, when AutoPK was combined with the LLaMA 3.1-70B model, it achieved an F1-score of 0.92 for half-life and 0.91 for clearance parameters. This was a substantial improvement over using LLaMA 3.1-70B directly, which performed 0.10 and 0.21 points lower, respectively. Even more remarkably, AutoPK transformed the performance of smaller, less powerful models. Models like Gemma 3-27B and Phi 3-12B, which struggled when used directly (with F1-scores often below 0.45 and hallucination rates as high as 60-95%), saw their F1-scores increase by 2 to 7 times with AutoPK. Gemma’s hallucination rates, for instance, plummeted from 60-95% down to a mere 8-14%.
A particularly noteworthy finding is that AutoPK enabled open-source models such as Gemma 3-27B to outperform commercial systems like GPT-4o Mini on several PK parameters. This highlights AutoPK’s ability to not only enhance powerful LLMs but also to elevate the capabilities of smaller or open-source models, making advanced PK data extraction more accessible.
The framework’s efficiency is also a key benefit. By filtering tables to retain only relevant rows, AutoPK reduces the amount of text an LLM needs to process. This lowers computational costs and speeds up extraction without losing critical context.
Future Directions
While AutoPK represents a significant leap forward, the researchers acknowledge areas for future improvement. The current system faces challenges with extremely complex and irregular table structures where crucial data (like drug, dosage, or species) are not in the same row as the PK parameter variants. Future work could explore more advanced, context-aware filtering, visual table parsing, or graph-based representations to handle these intricate layouts.
Additionally, expanding AutoPK to process less structured sources like PDFs and tables derived from optical character recognition (OCR) would broaden its applicability. Benchmarking against other state-of-the-art table understanding models, such as TAPAS, is also planned to further refine and validate the framework.
Also Read:
- Unlocking Drug Information: How AI Creates Knowledge Graphs from Leaflets
- EMR-AGENT: Intelligent Automation for Clinical Data Extraction
Conclusion
AutoPK offers a robust and scalable solution for extracting and standardizing pharmacokinetic data with high accuracy from complex scientific tables. By intelligently combining large language models with a hybrid similarity metric and validation steps, it effectively overcomes issues like structural variability and inconsistent terminology. This innovation promises to significantly streamline workflows in veterinary pharmacology, drug safety monitoring, and public health decision-making, ultimately supporting evidence-based decisions that protect public health. You can find more details about this research in the paper: AutoPK: Leveraging LLMs and a Hybrid Similarity Metric for Advanced Retrieval of Pharmacokinetic Data from Complex Tables and Documents.
