TLDR: STRUCT SENSE is a novel, task-agnostic, open-source framework that uses Large Language Models (LLMs) and a multi-agent system to extract structured information from unstructured scientific documents. It integrates domain-specific knowledge from ontologies, employs self-evaluative agents for iterative refinement, and includes human-in-the-loop mechanisms for quality assurance. Demonstrated in neuroscience tasks, STRUCT SENSE improves accuracy and adaptability, especially for complex information extraction, by leveraging human feedback to enhance LLM performance and semantic grounding.
The world of scientific research is expanding at an incredible pace, with millions of new articles published each year. This rapid growth, while exciting, has led to a significant challenge: information overload. Researchers find it increasingly difficult to keep up with the latest findings, evaluate their quality, and integrate new knowledge into their work. This is where the ability to extract structured information from unstructured sources, like free-text documents and scientific papers, becomes crucial for accelerating discovery and knowledge synthesis.
Traditional methods for extracting information often struggle with the specialized language and complex concepts found in scientific domains. While Large Language Models (LLMs) have shown great promise in various language tasks, their effectiveness can decrease in highly specialized fields that require deep domain knowledge. Furthermore, many existing LLM-based approaches are not easily transferable across different tasks or domains, limiting their usefulness and adaptability.
To tackle these challenges, a new framework called STRUCT SENSE has been introduced. It’s a modular, task-agnostic, and open-source system designed for structured information extraction, built upon the capabilities of LLMs. What makes STRUCT SENSE unique is its guidance by domain-specific knowledge, which is encoded in what are called ontologies. Ontologies provide a formal and explicit way to represent shared concepts and their relationships within a specific domain, helping the system understand complex content more effectively.
STRUCT SENSE also incorporates ‘agentic capabilities’ through self-evaluative judges. These judges create a feedback loop, allowing the system to iteratively refine its extractions. Crucially, it includes human-in-the-loop (HIL) mechanisms, ensuring quality control and validation by involving human experts in the process. This framework aims to overcome limitations related to domain sensitivity and the lack of generalizability across different tasks.
The architecture of STRUCT SENSE is composed of four specialized agents: the Extractor Agent, the Alignment Agent, the Judge Agent, and the Feedback Agent. The Extractor Agent converts raw text into structured data, typically in JSON format. The Alignment Agent then maps these extracted entities to their canonical counterparts in an ontology database, providing crucial domain knowledge grounding to prevent ambiguities (for example, distinguishing ‘cortex’ as a brain region from ‘cortex’ as a plant tissue). The Judge Agent evaluates the output from the Alignment Agent, assigning a confidence score. Finally, the Feedback Agent integrates human input, addressing potential inaccuracies or ‘hallucinations’ that LLMs might produce, thereby enhancing reliability.
Beyond these agents, STRUCT SENSE relies on an Ontology Database, which stores curated domain knowledge, and a Memory component that retains execution context across agents, enabling better collaboration and informed decision-making. The system is implemented in Python 3, leveraging the Crew.AI framework for its multi-agent architecture. It uses GROBID for document parsing and integrates with a vector database like Weaviate to store and retrieve ontology concepts, including a range of neuroscience-specific ontologies such as Uberon and Alzheimer’s Disease Ontology.
To demonstrate its utility, STRUCT SENSE was applied to three diverse neuroscience information extraction tasks: schema-based extraction for clinical assessments, metadata and resource extraction, and named entity recognition (NER) term extraction. These tasks showcased the framework’s ability to handle various data formats and structural goals, producing interpretable and FAIR-aligned (Findable, Accessible, Interoperable, and Reusable) outputs.
Evaluations were conducted both with and without human-in-the-loop intervention, using various LLMs including Claude 3.7 Sonnet, GPT-4o-mini, and DeepSeek V3 0324. The results consistently showed that human feedback improved performance, especially for more complex and open-ended tasks, and for models that initially performed less strongly. For instance, in named entity recognition, human intervention significantly boosted entity coverage and semantic grounding. While HIL generally increased operational costs and token usage, it proved invaluable for enhancing accuracy and semantic robustness, particularly for mid- and low-tier systems.
Also Read:
- Bridging the Gap: How Structured Memory Graphs Can Correct LLM Hallucinations
- AI-Powered Framework for Extracting Relationships from Data
The development of STRUCT SENSE represents a significant step towards more accurate, adaptable, and semantically enriched information extraction from scientific literature. For more in-depth information, you can refer to the full research paper.
