TLDR: LOGicalThought (LogT) is a new neurosymbolic framework that improves Large Language Models’ (LLMs) ability to perform high-assurance reasoning in critical domains like law and medicine. It does this by combining an LLM with an advanced logical language and reasoner to create a “dual context” (symbolic graph and logic program). This approach helps LLMs handle complex rules, exceptions, and non-monotonic logic, leading to significant performance improvements (11.84% average gain) and more robust, verifiable reasoning traces compared to traditional methods.
Large Language Models (LLMs) have shown incredible capabilities in understanding and generating human-like text. However, when it comes to critical fields like law and medicine, where accuracy, verifiability, and explicit evidence are paramount, LLMs often fall short. These “high-assurance” domains involve complex rules, statutes, and contracts, often with numerous exceptions that can change the meaning of a rule – a concept known as defeasible or non-monotonic logic. Traditional LLM methods, even advanced ones like Chain-of-Thought (CoT) prompting, struggle with the rigorous, verifiable reasoning required in these contexts.
A new neurosymbolic framework called LOGicalThought, or LogT, has been introduced to address these challenges. Developed by researchers from the University of Southern California and The University of Texas at Dallas, LogT aims to significantly enhance the high-assurance reasoning capabilities of LLMs. It does this by grounding natural language problems into a unique “dual neuro-symbolic context.”
How LOGicalThought Works
LogT transforms complex, unstructured reasoning tasks into a concise, ontologically grounded neurosymbolic representation. Unlike other approaches that might use a simple knowledge graph, LogT extracts both knowledge and logic from the raw natural language inputs. This dual context is crucial for navigating the subtleties, exceptions, and complex dependencies found in high-assurance guidelines.
The framework operates in three main stages:
1. Symbolic Context Generation: First, an LLM is used to select only the most relevant information from extensive guidelines based on a given scenario and hypothesis. From this filtered information, LogT creates a Symbolic Graph Context (C_sym). This includes an Ontology (O_rules) – a graph-based representation of rules and their relationships, Knowledge Triples (K_instance) – structured facts from the scenario and hypothesis, and Natural Language Queries (Q_nl) – precise questions derived from the hypothesis.
2. Logical Context Construction: Next, LogT synthesizes a formal Logic-based Context (C_log). An LLM translates the symbolic graph context into an ErgoAI logic program. ErgoAI is chosen for its robust support for higher-order logic and non-monotonic reasoning, which is essential for handling exceptions. This program includes facts derived from knowledge triples, rules and defeasible (overriding) rules from the ontology, and formal logic queries converted from the natural language questions. The generated program undergoes a two-stage verification process to correct syntactic errors and ensure compilability, resulting in a machine-readable knowledge base.
3. Hypothesis Evaluation: In the final stage, both the Symbolic Graph Context and the Logic-based Context are provided to an LLM. This “grounding” enables the LLM to perform a focused semantic evaluation of the pre-compiled knowledge against the original hypothesis. The LLM then predicts whether the hypothesis is entailed, contradicted, or neutral, and generates a detailed reasoning trace.
Also Read:
- Verifying LLM Reasoning with Proof-Carrying Chain-of-Thought
- SLogic: A New Approach to Interpretable Knowledge Graph Completion
Performance and Impact
LogT was evaluated on four multi-domain benchmarks, including ContractNLI (legal), Statutory Reasoning Assessment (SARA) (tax law), Safe Biomedical NLI for Clinical Trials (BioMedNLI) (medical), and a new Dungeons & Dragons NLI benchmark (games). Against four baselines, LogT consistently improved overall performance by an average of 11.84% across all LLMs. Smaller models, such as Mistral-7B and LLaMA-8B, showed the most significant gains.
The framework also demonstrated improved capabilities across all three critical reasoning modes: negation, implication, and defeasible reasoning. Gains were particularly pronounced in implication reasoning, with up to a 13.2% improvement on the D&D benchmark. Furthermore, LogT elicited approximately 21.5% more reasoning steps per example compared to standard Chain-of-Thought methods, with a notable increase in “apply rule” steps, indicating a more explicit, rule-based deduction process. The reasoning traces produced by LogT were also found to be more robust and better aligned with correct predictions.
An ablation study revealed that both the symbolic and logic-based contexts contribute to performance improvement, with the logic-based context providing the most substantial gains. This highlights the critical role of formal logic in achieving high-assurance reasoning.
This research marks a significant step towards making LLMs more reliable and trustworthy in domains where precise, verifiable reasoning is non-negotiable. For more in-depth information, you can read the full research paper here.
