TLDR: This research investigates whether Large Language Models (LLMs) can reliably generate commonsense knowledge to improve Natural Language Inference (NLI) tasks. The study found that while incorporating LLM-generated commonsense axioms doesn’t always boost overall accuracy, it significantly helps in identifying entailing relationships and moderately improves distinguishing contradictions and neutral cases. The paper also evaluates the factuality and consistency of these generated axioms, suggesting that pre-generated knowledge (P1) is more beneficial despite some over-generalization, compared to post-prediction explanations (P3) which often paraphrase the input.
Natural Language Inference (NLI), also known as Recognizing Textual Entailment (RTE), is a fundamental task in artificial intelligence that involves determining the semantic relationship between two sentences: a premise and a hypothesis. The goal is to figure out if the premise logically implies the hypothesis (Entailment), contradicts it (Contradiction), or if there’s no clear relationship (Neutral).
Human beings naturally use commonsense knowledge to make these inferences. For example, if the premise is “A woman is chatting as she drinks her coffee” and the hypothesis is “The woman is silent,” a person immediately understands this is a contradiction because chatting implies not being silent. This “commonsense knowledge” – practical information about how the world works, widely shared by most people – is crucial for NLI systems to emulate human-like reasoning.
However, a significant challenge in NLI has been the lack of comprehensive commonsense resources. Existing databases often don’t cover the vast array of situations encountered in premise-hypothesis pairs, leaving a “gap” in the knowledge required for accurate inference.
Leveraging Large Language Models for Commonsense
This research explores whether Large Language Models (LLMs), given their extensive training on vast amounts of text, can fill this gap by generating the necessary commonsense knowledge. The study investigates two key questions: first, how reliably can LLMs generate these commonsense “axioms” (basic, self-evident principles), and second, what impact does incorporating this generated knowledge have on the accuracy of NLI predictions?
To answer these questions, the researchers used the Llama-3.1-70b-Instruct model and two well-known NLI datasets: SNLI (Stanford Natural Language Inference) and ANLI (Adversarial NLI). They designed three types of prompts to test different scenarios:
- Prompt 1 (P1): Used to generate commonsense axioms for a given premise-hypothesis pair before making an inference prediction.
- Prompt 2 (P2): Takes the commonsense axioms generated by P1 and appends them to the premise-hypothesis pair to see how this external knowledge affects the LLM’s inference prediction.
- Prompt 3 (P3): Serves as a baseline, asking the model to predict the inference label without any external commonsense axioms, but requiring a short explanation of its reasoning. This represents the model’s implicit understanding.
Evaluating Reliability and Impact
The study introduced modified evaluation criteria to assess the reliability of the LLM-generated commonsense axioms. They looked at:
- Factuality: How accurate and helpful are the generated axioms for predicting the correct inference label? They used another LLM, Claude-3.5-Sonnet, as a judge to rate helpfulness.
- Consistency: How consistently does the LLM generate similar commonsense axioms for the same premise-hypothesis pair across multiple runs? Again, Claude-3.5-Sonnet was used to rate similarity.
- Prediction Error Analysis: How do the generated axioms affect the model’s ability to correctly classify entailment, contradiction, and neutral relationships compared to the baseline?
Also Read:
- Unpacking LLM Performance in SPARQL Query Generation: The Role of Knowledge and Memorization
- Unpacking How Question Types Affect Large Language Model Performance
Key Findings
The results showed several interesting insights. While incorporating commonsense knowledge didn’t consistently improve overall accuracy across all scenarios, it significantly helped the model distinguish “entailing” instances. It also moderately improved the identification of “contradictory” and “neutral” inferences.
Specifically, the axioms generated before prediction (P1) were found to be more consistent across most inference classes in the SNLI dataset and showed higher factuality for entailments in the ANLI dataset, compared to axioms generated after prediction (P3). However, P3-generated axioms often appeared more “factual” overall because they tended to paraphrase the input rather than generate new, distinct commonsense knowledge. P1-style axioms, despite sometimes over-generalizing, were structured more like true commonsense statements.
A notable benefit was observed in handling spatial contradictions. For instance, if a premise states “Man in white sweater looks from his press box onto the pitch” and the hypothesis is “A man is sits on a casino,” the P1-generated axiom correctly identified the contradiction (a person cannot be in two locations simultaneously). In contrast, P3 often considered these unrelated events.
However, a limitation was that the model sometimes over-generalized with P1 axioms, leading to misclassifications. For example, “A man playing an electric guitar on stage” and “A man is performing for cash” might lead P1 to generate an axiom about performers often getting paid, causing an incorrect “Entailment” prediction, whereas P3 would correctly identify it as “Neutral” due to uncertainty.
In conclusion, this study highlights the significant potential of LLMs to generate commonsense knowledge for NLI tasks, particularly in improving the identification of entailment. It also underscores the importance of ensuring the quality and accuracy of this generated knowledge for overall performance gains. Future work aims to refine the axiom generation process to produce even more factual and consistent commonsense knowledge. You can read the full research paper for more technical details and results at this link.
