TLDR: This research evaluates fine-tuning four decoder-only Large Language Models (LLaMA-3.2, Phi-3.5, DeepSeek-R1-LLaMA-Distilled, Gemma-2) for Abstract Meaning Representation (AMR) parsing. It demonstrates that straightforward fine-tuning can achieve performance comparable to complex State-of-the-Art AMR parsers, with LLaMA-3.2 showing strong semantic understanding and Phi-3.5 excelling in structural validity. The study validates these findings across different graph complexities and text domains, suggesting LLMs are highly effective for deep semantic parsing.
A recent research paper explores how Large Language Models (LLMs) can be fine-tuned to understand and represent the meaning of sentences, a task known as Abstract Meaning Representation (AMR) parsing. AMR is a way to encode sentence meaning as a graph, where concepts are nodes and relationships between them are edges. This allows computers to grasp the underlying meaning of language, moving beyond just surface-level text.
Traditionally, AMR parsing has required specialized and complex systems, making it difficult to implement. However, this paper demonstrates that a simpler approach – fine-tuning general-purpose LLMs – can achieve comparable or even better accuracy while significantly reducing complexity. The study focused on four different decoder-only LLM architectures: Phi-3.5, Gemma-2, LLaMA-3.2, and DeepSeek-R1-LLaMA-Distilled. These models were fine-tuned using a technique called LoRA (Low-Rank Adaptation), which makes the process more efficient and allows for training on more accessible hardware like consumer-grade GPUs.
The researchers used the LDC2020T02 Gold AMR3.0 dataset for training and evaluation. This dataset contains nearly 60,000 English sentences, each paired with its corresponding AMR graph. The models were configured for efficient loading, using 4-bit quantization to reduce memory usage and bfloat16 for numerical precision, enabling them to handle complex AMR structures.
The evaluation used the SMATCH metric, a standard for comparing AMR graphs by measuring the overlap between predicted and reference graphs. The results were quite impressive. LLaMA-3.2, one of the models tested, achieved an F1 score of 0.804 on the LDC2020T02 test set. This performance is on par with some highly specialized State-of-the-Art (SOTA) AMR parsers, such as APT + Silver (IBM), which also scored 0.804. While slightly below the absolute top SOTA models, the fact that general-purpose LLMs can achieve such results with straightforward fine-tuning is a significant finding.
Further analysis delved into the models’ performance based on the “depth” of the AMR graphs, which indicates semantic complexity. LLaMA-3.2 consistently showed strong semantic performance, while Phi-3.5 excelled in maintaining structural validity, meaning it produced more grammatically correct and well-formed AMR graphs. DeepSeek-R1-LLaMA-Distilled offered a good balance between semantic accuracy and structural correctness.
To validate these findings at a larger scale, the models were also tested on a “silver” dataset, which is much larger and generated using advanced ensemble methods. On this dataset, LLaMA-3.2 again led in semantic performance. Interestingly, at the highest levels of complexity, all models showed a convergence in their F1 scores, suggesting a potential performance ceiling for current transformer-based approaches when dealing with extremely intricate semantic structures.
The study also examined how well the models performed across different types of text, from discussion forums to news articles. It found that performance varied by domain, with formal content like “Proxy Reports” generally yielding higher scores. DeepSeek-R1-LLaMA-Distilled showed the most consistent performance across these diverse text domains, possibly due to its Chain-of-Thought reasoning capabilities.
Also Read:
- AI’s New Frontier: Classifying Sentences to Master Literature Reviews
- Simplifying Spanish: How Large Language Models Improve Text Readability
In conclusion, this research highlights the significant potential of fine-tuning decoder-only LLMs for AMR parsing. It demonstrates that these models can achieve performance comparable to specialized systems with simpler implementation, paving the way for more robust and accessible NLP systems that can truly understand the meaning behind our words. For more details, you can read the full research paper here: Evaluation of Large Language Model Architectures in Abstract Meaning Representation Parsing.
