TLDR: A study evaluated four large language models (OpenAI GPT-4.1 Mini, Gemini 2.0 Flash, Meta Llama 3 70B, and DeepSeek R1) on their ability to answer authentic legal questions in the Bengali legal context. Using a dual evaluation framework involving both an LLM-as-judge and licensed Bangladeshi legal professionals, the research assessed AI responses for factual accuracy, legal appropriateness, completeness, and clarity. While AI models showed potential for providing informative answers, human lawyers found OpenAI 4.1 Mini and Gemini 2.0 Flash to be top performers, with OpenAI 4.1 Mini demonstrating superiority in factual accuracy, completeness, and clarity. The study highlights the critical need for expert validation and safeguards to address issues like misinformation and context-specific reasoning before AI can be safely deployed for legal assistance in Bangladesh.
Bangladesh faces a significant challenge in providing its citizens with accessible legal assistance. Millions are left without proper legal recourse due to a severe shortage of lawyers, prohibitively high fees, and a complex, multi-layered legal system that combines common law, Islamic law, and customary practices. This intricate landscape makes it difficult for ordinary citizens to navigate legal procedures and understand archaic terminology.
In this context, the emergence of generative Artificial Intelligence (AI) models, such as OpenAI’s GPT-4.1 Mini, Google’s Gemini 2.0 Flash, Meta’s Llama 3 70B, and DeepSeek R1, presents a potential avenue to democratize legal aid. These AI systems could offer 24/7, low-cost, and anonymous legal advice, reaching remote areas and overcoming financial and social barriers that currently prevent many from seeking help.
However, deploying AI in legal settings carries substantial risks. Unlike other domains where AI errors might cause minor inconvenience, incorrect legal guidance can lead to severe real-world consequences, including financial loss, criminal liability, or loss of legal rights. Previous studies have highlighted troubling patterns in AI-generated legal content, such as the fabrication of case citations, misstatement of legal procedures, and the provision of outdated or jurisdiction-inappropriate advice. These “hallucinations” are particularly dangerous as they appear authoritative and are difficult for non-experts to verify.
A recent study, titled Assessing the Reliability of Large Language Models in the Bengali Legal Context: A Comparative Evaluation Using LLM-as-Judge and Legal Experts, aimed to address these critical gaps by evaluating AI performance on real-world Bangladeshi legal questions. Unlike prior research that often relied on synthetic scenarios, this study utilized 250 authentic legal queries collected from “Know Your Rights,” a popular Facebook group where Bangladeshi citizens seek legal advice from verified professionals.
The research employed a comprehensive dual evaluation framework. Initially, a state-of-the-art LLM (Gemini 2.5 Pro) served as a judge, assessing AI-generated responses across four key dimensions: factual accuracy, completeness, conciseness, and clarity. Following this, the same set of questions was evaluated by three licensed Bangladeshi legal professionals, each with a minimum of five years of experience, using the same criteria. Automated evaluation metrics like BLEU scores were also applied, but these traditional metrics were found to be insufficient for capturing the nuances of complex legal reasoning.
The automatic metric-based evaluation revealed minimal lexical and n-gram overlap between AI-generated answers and reference answers provided by professional lawyers. This indicated that while AI responses were often structured and detailed, they differed significantly in phrasing from the informal, practical guidance typically offered by human experts on social media. This discrepancy underscored the limitations of traditional metrics in assessing complex legal advice.
When an LLM was used as a judge, Gemini 2.0 Flash emerged as the top performer, achieving the highest average score (3.772), closely followed by OpenAI 4.1 Mini (3.612). Statistical analysis confirmed significant differences between Gemini 2.0 Flash and Meta Llama 3 (70B) and DeepSeek R1, as well as between Meta Llama 3 (70B) and OpenAI 4.1 Mini. However, no statistically significant difference was found between the top-performing Gemini 2.0 Flash and OpenAI 4.1 Mini, suggesting they were statistically on par.
The professional lawyer evaluation, however, presented a slightly different hierarchy. OpenAI 4.1 Mini received the highest average score (4.25), with Gemini 2.0 Flash ranking second (3.96). This human assessment confirmed that OpenAI 4.1 Mini and Gemini 2.0 Flash formed a top-tier, statistically superior to Meta Llama 3 (70B) and DeepSeek R1. Similar to the LLM-as-judge evaluation, the lawyers found no statistically significant difference between OpenAI 4.1 Mini and Gemini 2.0 Flash.
A deeper comparison between the LLM-as-judge and professional lawyer evaluations revealed a notable discrepancy in how they perceived model performance. While the LLM-as-judge found no significant overall difference between OpenAI 4.1 Mini and Gemini 2.0 Flash, the lawyers rated OpenAI 4.1 Mini as significantly superior in factual accuracy, completeness, and clarity. Gemini 2.0 Flash, on the other hand, was rated significantly better in conciseness by the human experts. This highlights that while both models are top performers, human legal experts identified specific strengths in OpenAI 4.1 Mini that the LLM-as-judge did not fully capture.
The study acknowledges several limitations, including potential evaluation bias due to the differing response formats (structured AI vs. informal human comments) and the dataset being sourced from a single online platform. Despite these, the findings underscore that while AI models can provide informative and coherent answers, significant gaps remain in legal accuracy and context-specific reasoning, particularly in a pluralistic legal system like Bangladesh’s.
Also Read:
- AI Breakthrough: Automating the Extraction of Criminal Facts from Court Opinions
- Geographic Disparities in AI’s Legal Knowledge: A Study on LLM Hallucinations
In conclusion, the research emphasizes the critical need for rigorous expert validation and comprehensive safeguards before AI systems can be safely deployed for legal consultation. While AI has the potential to serve as a valuable legal assistant, providing preliminary guidance, it cannot yet replace human practitioners. Future work could focus on fine-tuning models with jurisdiction-specific datasets and integrating advanced techniques like Retrieval-Augmented Generation (RAG) to enhance factual reliability and ensure responsible AI deployment in sensitive domains like law.
