TLDR: This paper introduces an LLM-assisted workflow for automated ontology generation from software engineering standards. It focuses on relation triplet extraction, using open-source LLMs like Mistral-7B to build an ontology scaffold. The method is evaluated against manual benchmarks and shows competitive performance, often achieving higher precision than traditional OpenIE methods, paving the way for more structured knowledge representation in software engineering.
In the realm of software engineering, standards are crucial, but their vast, unstructured text can be a challenge to navigate and utilize effectively. Imagine a system that could automatically read these lengthy documents and transform their inherent knowledge into a structured, machine-readable format. This is precisely the goal of a recent research paper titled “LLM-based Triplet Extraction for Automated Ontology Generation in Software Engineering Standards” by Songhui Yue.
The paper introduces an innovative approach to Automated Ontology Generation (AOG) specifically tailored for Software Engineering Standards (SES). Ontologies, which are formal representations of knowledge, have long been vital for knowledge representation and reasoning. However, their manual creation is a laborious process, and scaling their use requires automation.
The Challenge of Unstructured Text
Software engineering standards are often characterized by long, complex, and noisy text, filled with domain-specific terms. Extracting meaningful relationships from such text is the first critical step towards building an ontology. While Large Language Models (LLMs) have shown promise in generating ontology artifacts, simply prompting them faces several hurdles: reproducibility issues, difficulty in scaling to very long or multimodal documents, and the need for a robust engineering workflow for complex, evolving tasks.
A Novel LLM-Assisted Workflow
Instead of relying solely on simple prompt engineering, this research proposes an LLM-assisted approach that integrates LLMs as a powerful aid within a structured AOG workflow. The process includes several key stages: document segmentation, mining candidate terms, LLM-based relation inference, term normalization, and cross-section alignment. This assertion-led approach prioritizes extracting instance-level assertions (triples) and schema-level candidates, which can later be validated and consolidated into a formal ontology.
The methodology focuses on building an “ontology scaffold” by processing the text sentence-by-sentence. For each sentence, tools like spaCy are used to identify noun phrases and verbs, which become candidate entities and relation vocabularies. An open-source LLM, specifically Mistral-7B, is then employed to infer relations and generate subject-predicate-object triples in a structured JSON format. This method ensures a stable output format, crucial for the overall success of the automation.
Evaluation and Key Findings
To rigorously evaluate the proposed method, the study constructed three “gold-standard” reference sets of varying granularities from the “Software Engineering Code of Ethics and Professional Practice” (SECEPP) – a publicly available standard chosen for its conciseness and relevance. The performance of the LLM-based system (Pred-LLM) was compared against a traditional sentence-level OpenIE baseline (Pred-openIE).
The results are promising. The Pred-LLM system demonstrated competitive, and often superior, performance compared to OpenIE. It consistently yielded higher precision for both nodes (terms) and triples (relationships) across various similarity thresholds. While recall on very dense gold sets remains an area for improvement, the study found that a 7B open-source LLM is already a strong contender, capable of generating highly trustworthy triple matches, especially at higher similarity thresholds.
Also Read:
- Making Sense of Privacy Policies: An AI-Powered Approach
- ReCode: Enhancing AI’s Code Repair Capabilities with Smart Retrieval
Looking Ahead
This research marks a significant step towards fully automated ontology generation for software engineering standards. The proposed workflow and preliminary results highlight the potential of open-source LLMs in transforming unstructured textual data into valuable, structured knowledge. Future work will focus on enhancing recall, extending the method to longer and more diverse SES documents, and ultimately producing a formal OWL 2 ontology and public benchmarks. This advancement could greatly benefit automated software engineering, ensuring that intelligent agents comply with ethical and professional practices implicitly encoded in these standards. For more details, you can refer to the full research paper here.
