TLDR: A study comparing small (7-8B parameters) and large (1-2T parameters) language models for requirements classification found that while LLMs had a 2% higher average F1 score, this difference was not statistically significant. SLMs performed comparably, sometimes even outperforming LLMs in recall, and offer significant advantages in privacy, cost, and local deployability, suggesting they are a viable alternative for this task.
In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) have garnered significant attention for their impressive capabilities across various natural language processing (NLP) tasks. However, their widespread adoption, especially in sensitive domains like requirements engineering (RE), is often hampered by concerns regarding high computational costs, potential data-sharing risks, and reliance on external cloud services. This has led researchers to explore the viability of small language models (SLMs) as a more lightweight, cost-effective, and locally deployable alternative.
A recent study titled “Does Model Size Matter? A Comparison of Small and Large Language Models for Requirements Classification” by Mohammad Amin Zadenoori, Vincenzo De Martino, Jacek Dąbrowski, Xavier Franch, and Alessio Ferrari, delves into this crucial question. The research aimed to clarify how well SLMs perform compared to their larger counterparts in the specific context of requirements classification – a fundamental task in RE that involves categorizing requirements into types like functional, non-functional, security, or performance.
The Study’s Approach
The researchers conducted a preliminary study comparing eight different language models: three LLMs (GPT-5, Grok-4, and Claude-4) and five SLMs (Qwen2-7B, Falcon-7B, Granite-3.2-8B, Ministral-8B, and Llama-3-8B). While the LLMs typically boast trillions of parameters, the SLMs evaluated ranged from 7 to 8 billion parameters, making them up to 300 times smaller. These models were put to the test on three public datasets commonly used for requirements classification: PROMISE, PROMISE Reclass, and SecReq. The evaluation focused on standard metrics such as precision, recall, and F1 score, using a Chain-of-Thought plus Few-Shot prompting strategy.
Surprising Findings: Size Isn’t Everything
The core finding of the study challenges the intuitive assumption that bigger models always mean better performance. While LLMs did achieve an average F1 score that was 2% higher than SLMs across all tasks, this difference was not found to be statistically significant. This suggests that, from a statistical standpoint, the performance gap is not substantial enough to definitively declare LLMs superior for this specific task.
Even more remarkably, SLMs demonstrated highly competitive performance, almost matching LLMs across all datasets. In some instances, SLMs even outperformed LLMs. For example, on the PROMISE Reclass dataset, SLMs like Qwen2-7B and Falcon3-7B showed a significant advantage in recall, indicating their exceptional effectiveness at identifying all relevant instances and minimizing false negatives. This highlights specialized strengths within SLMs that could be crucial for scenarios where certain types of errors are more critical.
Another significant insight from the study was that the characteristics of the dataset itself played a more substantial role in determining performance than the model size. This implies that the nature of the requirements and their classification categories can influence a model’s effectiveness more than whether it’s a small or large model.
Also Read:
- Bridging the Gap: How Symbolic AI Enhances Transparency and Reasoning in Large Language Models
- Enhancing LLM Knowledge Distillation with Counterfactual Explanations in Low-Data Settings
Implications and Future Directions
The study’s conclusions offer compelling evidence that SLMs are a valid and powerful alternative to LLMs for requirements classification. Their advantages in terms of privacy (due to local deployability), reduced computational cost, and resource efficiency make them particularly attractive for companies dealing with confidential data or operating with limited resources. While the researchers acknowledge that the limited sample size might have led to a Type II error (failing to detect a true difference), they argue that even a 2% performance difference is acceptable given the significant benefits SLMs offer.
Looking ahead, the research outlines several promising avenues for future work. These include exploring the explainability of model classifications, extending evaluations to other requirements engineering tasks like traceability and model generation, quantifying the energy consumption differences between SLMs and LLMs, and systematically studying their execution speeds. Ultimately, the goal is to develop tools that allow practitioners to flexibly choose between locally hosted SLMs and API-based LLMs, optimizing for specific constraints like data privacy and computational resources.
