TLDR: A pilot study compared traditional NLP tools (NLTK, spaCy, Stanza) and large language models (Gemini, DeepSeek, Qwen) on a small, ambiguity-rich dataset for Named Entity Recognition. LLMs generally performed better on context-sensitive entities like person names, with Gemini achieving the highest F1-score. Traditional tools, particularly Stanza, showed stronger consistency in structured entities like locations and dates. The study highlights that while LLMs offer improved contextual understanding, traditional tools remain competitive for specific, structured tasks, influencing model selection based on task requirements and cost.
Named Entity Recognition (NER) is a foundational task in Natural Language Processing (NLP) that involves identifying and classifying key information—like people, places, organizations, dates, and times—within unstructured text. It’s a crucial component for many modern language technologies, from information extraction to semantic search.
Over the years, NER systems have evolved significantly, moving from rule-based and statistical models to advanced deep learning and transformer-based architectures. Recently, the emergence of large language models (LLMs) like Gemini, DeepSeek, and Qwen has opened new avenues for more context-aware and flexible entity recognition, challenging the established performance of traditional NLP tools such as NLTK, spaCy, and Stanza.
Comparing Approaches: LLMs vs. Traditional Tools
A recent pilot study, detailed in the paper “Is ‘Hope’ a Person or an Idea? A Pilot Benchmark for NER”, set out to compare these two categories of NER systems. The research focused on understanding the performance patterns that emerge when traditional NLP libraries are pitted against LLMs on a small, carefully annotated dataset rich in ambiguous entities. The goal was to inform task-specific model selection for practitioners.
The study utilized a custom dataset of 119 tokens, manually annotated across five entity types: PERSON, LOCATION, ORGANIZATION, DATE, and TIME. This dataset was specifically designed to include challenging scenarios such as ambiguous person names (e.g., ‘Justice Hope’), multi-word entities, and context-sensitive temporal expressions (e.g., ‘midday’, ‘dusk’). The six systems evaluated included NLTK, spaCy, and Stanza from the traditional NLP toolset, and Gemini-1.5-flash, DeepSeek-V3, and Qwen-3-4B representing the LLM category.
Key Findings: Contextual Understanding vs. Consistency
The evaluation, primarily using the F1-score, revealed several interesting insights:
- LLMs generally outperformed traditional tools in recognizing context-sensitive entities, particularly person names. Gemini-1.5-flash achieved the highest overall average F1-score (0.824) and a near-perfect score for PERSON entities (0.960), accurately disambiguating names like ‘Justice Hope’. DeepSeek-V3 also matched Gemini’s high performance on PERSON and LOCATION.
- Traditional systems showed greater consistency in structured tags. Stanza, a deep learning-based toolkit, demonstrated robust performance across all categories, excelling in ORGANIZATION (0.846), LOCATION (0.857), and DATE (0.857). spaCy also performed exceptionally well in DATE recognition (0.933).
- Variability among LLMs was observed, especially in handling temporal expressions and multi-word organizations. While LLMs showed strength in PERSON entities, their performance on ORGANIZATION and TIME categories was sometimes less consistent than traditional tools. Qwen-3-4B, for instance, struggled significantly with DATE entities.
- Both groups of models faced challenges with TIME expressions, indicating room for improvement in handling time-of-day references like ‘dusk’ or ‘midday’.
Implications for Model Selection
The study concludes that while LLMs offer improved contextual understanding, making them highly effective for ambiguous or context-dependent entities, traditional tools remain competitive and often more consistent in structured tagging tasks. For applications requiring high-volume processing of dictionary-driven spans, where determinism and speed are paramount, lighter traditional libraries like Stanza might still be the more practical and cost-effective choice. LLMs, with their higher per-query costs, are best justified when inputs are rich in ambiguous names and recall is a critical factor.
Also Read:
- Evaluating Semantic Similarity: Uncovering Flaws and Finding Better Measures for Code and Text
- Evaluating AI Agent Architectures for Enterprise Workflows
Looking Ahead
The researchers acknowledge the pilot nature of the study, citing limitations such as the small dataset size and single annotator. Future work aims to expand the dataset, involve multiple annotators for increased reliability, and conduct more extensive sensitivity analyses on LLM prompts and outputs to further refine our understanding of these powerful language models.
