TLDR: A new research paper introduces a method to build Legal Knowledge Graphs (LKGs) from Japanese court judgments. These graphs explicitly map the complex reasoning paths from facts to legal provisions, addressing limitations of large language models in legal contexts. The LKG approach significantly improves the accuracy of retrieving relevant legal provisions based on factual inputs, demonstrating the value of structured knowledge for transparent and interpretable legal AI.
Understanding how courts interpret and apply laws to specific situations is crucial for legal professionals and the public alike. Court judgments are rich sources of this information, but the intricate web of legal reasoning—connecting facts, legal rules, and their application—is often implicit and challenging for automated systems to grasp. Traditional approaches, including advanced large language models (LLMs), frequently struggle with identifying the correct legal context, accurately tracing the relationship between facts and legal norms, and representing the layered structure of judicial thought.
A recent research paper, titled Capturing Legal Reasoning Paths from Facts to Law in Court Judgments using Knowledge Graphs, addresses these challenges by proposing a novel method to construct Legal Knowledge Graphs (LKGs) from court decisions. This approach aims to make the complex structure of legal reasoning explicit and machine-readable, offering a more robust foundation for legal AI systems.
The Problem with Current AI in Law
While large language models can generate plausible legal text, they face significant hurdles in reliable legal reasoning. They often lack the ability to establish a legal framework, meaning they can’t always determine the correct jurisdiction or legal system for a given case. For instance, the legal defense for defamation can vary dramatically between countries. LLMs also struggle with structural grounding, failing to connect facts, norms, and provisions through explicit reasoning paths, often producing outputs that mimic surface-level patterns without truly understanding the underlying logic. Furthermore, courts can adopt multi-layered or even conflicting perspectives, which LLMs find difficult to summarize quantitatively.
Even retrieval-augmented generation (RAG) techniques, which combine LLMs with external document retrieval, fall short. While RAG improves factual grounding, it doesn’t reconstruct the underlying logic of judicial decision-making or model the structured inferential processes that link legal elements. This highlights the ongoing need for explicit structured representations like LKGs.
Building a Legal Knowledge Graph
The researchers constructed an LKG from 648 Japanese administrative court decisions. Their method involves a three-step process:
1. Schema Design: They defined a legal ontology that captures the core reasoning structure. Key classes include “Fact,” “LegalNorm,” “LegalApplication,” and “Provision.” Crucially, “LegalApplication” represents an explicit reasoning step that connects a legal norm to a fact. The schema also supports hierarchical reasoning, allowing connections within the same category (e.g., Fact to Fact) to model layered legal interpretations.
2. Node and Edge Extraction: Using GPT-4o, the system extracts key elements like evidence, facts, laws, and legal interpretations from court judgments. To improve consistency and accuracy, a case overview is included in the prompt, and a fictional “Martian Law” scenario is used for few-shot examples to prevent overfitting to real legal data. Statutory references are also normalized. Edges, representing relationships between these nodes, are constructed using tailored prompts. For instance, “Provision → Legal Norm” links are identified based on local proximity, while “Legal Norm → Legal Application” and “Fact → Legal Application” links use a scoped-history prompting strategy to capture connections that might span distant sections of a document.
3. Legal Search with LKG: The constructed LKG enables a new type of legal search. Instead of starting with legal provisions, the system begins with observed facts. Each fact node is embedded and indexed, allowing the system to retrieve the most similar facts from other cases and, subsequently, the legal provisions linked to them through the graph’s reasoning paths.
Also Read:
- Integrating Knowledge Graphs for Advanced Multi-hop Question Answering in Language Models
- GLARE: A New Approach to Predicting Legal Outcomes
Evaluation and Impact
The evaluation of the LKG focused on three aspects:
- Extraction Accuracy: Legal experts assessed the accuracy of node and edge extractions, finding high precision and recall for most categories.
- Structural Coherence: The LKG’s overall structure was analyzed, revealing properties like hierarchical composition and element reuse, consistent with how legal arguments unfold.
- Legal Search Performance: The LKG-based retrieval method significantly outperformed several GPT-4o-based baselines (Simple, With Context, and RAG) in both macro and micro recall for identifying relevant statutory articles from factual inputs. This demonstrates that structured legal knowledge leads to more accurate and consistent reasoning than language models alone, even when they have access to relevant source content.
The qualitative analysis further showed that the LKG effectively captures judicial logic, from straightforward cases to more nuanced scenarios involving factual variations or interactions with constitutional principles. For example, it could trace how minor factual differences (like distance from a project site) could lead to different legal outcomes under the same norm.
In conclusion, this research highlights the essential role of explicitly grounded knowledge structures in developing interpretable and generalizable legal AI systems. By making legal reasoning paths explicit and machine-interpretable, the LKG approach offers a powerful tool for legal search and understanding, moving beyond surface-level textual patterns to capture the true logic of judicial decisions.
