TLDR: A study evaluated different AI models, including large language models (LLMs) and fine-tuned transformers, for improving the accuracy of police-reported crash data by identifying secondary crashes from narratives in Kentucky. Fine-tuned transformer models like RoBERTa showed the best balance of accuracy and efficiency, outperforming LLMs in speed while achieving slightly better F1-scores. The research provides practical recommendations for deploying AI to enhance crash data quality, emphasizing local deployment for privacy and incremental processing for scalability.
High-quality crash data is fundamental for effective traffic safety analysis, yet police-reported databases frequently suffer from issues like underreporting and miscoding, particularly concerning secondary crashes. These inaccuracies can undermine efforts to identify crash hotspots, understand causal relationships, and prioritize safety investments.
Crash narratives, which are detailed descriptions written by police officers, offer a valuable resource to cross-check and enhance coded crash data. Traditionally, traffic engineers manually review these reports, a process that is labor-intensive and prone to inconsistencies. Early machine learning attempts to automate this review, using methods like ‘bag-of-words’ text classification, often resulted in high rates of false positives and negatives due to their inability to capture linguistic context.
Recent advancements in natural language processing (NLP), especially with transformer-based large language models (LLMs), have opened new avenues for mining crash narratives. These models, pre-trained on vast text databases, excel at understanding nuanced syntax and semantics, making them capable of processing large volumes of crash narratives to extract insights that traditional methods often miss.
This study addresses a critical gap by providing a comprehensive evaluation of narrative-mining algorithms specifically for enhancing crash data quality, using secondary crash identification in Kentucky as a case study. Researchers compared three main approaches: zero-shot open-source LLMs (like LLaMA3:70B, DeepSeek-R1:70B, Qwen3:32B, Gemma3:27B), fine-tuned transformer models (such as BERT, DistilBERT, RoBERTa, XLNet, Longformer), and traditional logistic regression as a baseline.
The study utilized a large dataset of 16,656 manually reviewed crash narratives from Kentucky between 2015 and 2022, with 3,803 confirmed secondary crashes. Models were trained on data from 2015-2021 and tested on 1,771 narratives from 2022.
Key Findings
The results showed that fine-tuned transformer models generally outperformed the larger LLMs operating in a zero-shot inference mode for this specific classification task. RoBERTa emerged as the top performer, achieving an F1-score of 0.90 and an accuracy of over 95%. BERT and DistilBERT also performed exceptionally well, with F1-scores between 0.88 and 0.89 and accuracies of 94-95%. These models demonstrated an excellent balance of precision (correctly identifying secondary crashes) and recall (catching most true secondary crashes).
Zero-shot LLMs like LLaMA3:70B and DeepSeek-R1:70B showed strong performance with F1-scores of 0.85-0.86, but they were marginally behind the fine-tuned models. A significant difference was observed in computational efficiency: LLMs required orders of magnitude more time to process the test set (e.g., LLaMA3:70B took 139 minutes, DeepSeek-R1:70B over 12 hours) compared to fine-tuned models, which processed the same test set in mere seconds after a brief training period (e.g., RoBERTa trained in 13 minutes and tested in 8 seconds).
The traditional logistic regression baseline lagged significantly, with an F1-score of 0.66 and 83% accuracy, highlighting the substantial improvements offered by more advanced NLP models.
Also Read:
- Unraveling Why AI Reasoning Models Struggle with Complex Multi-Hop Questions
- SymbolBench: Assessing Large Language Models in Time Series Reasoning
Practical Implications for Deployment
The study offers several practical recommendations for transportation agencies. For secondary crash detection, where underreporting is a major concern, models with high recall are desirable. Fine-tuned transformer models like RoBERTa are recommended due to their balance of high precision and recall, and their applicability extends to other crash types like alcohol involvement or wrong-way driving.
An ensemble strategy, combining different models (e.g., RoBERTa as a primary checker with XLNet and Gemma for parallel analysis), could further reduce human workload by flagging disagreements for manual review. While zero-shot LLMs are computationally intensive, their versatility in cross-checking narratives against multiple coded fields without task-specific retraining makes them valuable as generalist auditors of crash data.
Privacy is paramount given the sensitive information in crash narratives. Therefore, on-premise or local deployment of these models is strongly recommended to keep data under agency control, rather than relying on online or API-based services. The trend towards smaller, yet capable, LLMs and techniques like model quantization will facilitate local deployments.
Regarding computational runtime, fine-tuned transformer models can easily scale to large datasets, processing a year’s worth of crash narratives in minutes. For LLMs, an incremental workflow—processing new crash reports as they arrive (e.g., weekly or monthly batches)—can mitigate the challenge of their slower processing times. This approach is feasible as crash data quality verification typically does not require real-time processing.
In conclusion, deploying an LLM-assisted crash data verification system is feasible and promises to become even more practical as technology advances. Agencies should plan for solutions that leverage LLMs’ capabilities while addressing current limitations through thoughtful deployment strategies: ensuring data privacy, adopting incremental processing, using a mix of models with human oversight, and balancing hardware constraints with model capabilities. This will significantly enhance crash data quality and analytical efficiency, leading to safer and more informed transportation safety management. For more details, you can refer to the full research paper here.
