From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text
1University of Science, Vietnam National University Ho Chi Minh City (VNU-HCM) · SELab
FISU Joint Conference on Artificial Intelligence (FJCAI 2026), Vietnam & arXiv (cs.CL)
Figure 1: Figure 1: Error distribution across 480 outputs using the 9-category Legal Reasoning Error Typology.
Highest Accuracy score yet highest error count (41 Misinterpretations, 39 Incorrect Examples)
Top Readability (4.93) and Consistency (4.28) with zero Omission of Core Elements or Oversimplification
Strips intricate statutory conditions into overly brief maxims (3.78 Accuracy)
Paper Abstract
The complexity of Vietnam's legal texts presents a significant barrier to public access to justice. While Large Language Models offer a promising solution for legal text simplification, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. This paper introduces a comprehensive dual-aspect evaluation framework to address this need. First, we establish a performance benchmark for four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three key dimensions: Accuracy, Readability, and Consistency. Second, to understand the 'why' behind these performance scores, we conduct a large-scale error analysis on a curated dataset of 60 complex Vietnamese legal articles, using a novel, expert-validated error typology. Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy, while models like Claude 3 Opus achieve high Accuracy scores that mask a significant number of subtle but critical reasoning errors.
Research Motivation
Deploying LLMs to simplify codified civil law (such as Vietnam's Penal Code, Civil Code, and Land Law) risks generating fluent, plausible explanations that omit statutory exceptions or misapply legal rules in examples—failures that aggregate Likert accuracy scores completely conceal.
Guiding Research Questions
- RQ1How do frontier LLMs (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, Grok-1) trade off legal accuracy against layperson readability and multi-run consistency on complex Vietnamese statutes?
- RQ2What specific reasoning failure modes emerge when models are required to both explain a statute and generate a concrete legal application example?
Methodology & Architecture
We curate 60 complex Vietnamese legal articles (20 Penal Code 2015, 20 Civil Code 2015, 20 Land Law 2024) characterized by multiple exceptions and sequential logic, generating 480 outputs (4 models × 60 articles × 2 runs at T=0.2). Phase 1 benchmarks Legal Accuracy (rated by 5 trained law evaluators), Readability (rated by 253 lay participants), and Consistency (semantic MiniLM cosine + BLEU + length + human rating). Phase 2 applies a novel 9-category Legal Reasoning Error Typology validated with legal experts from Hanoi Open University, achieving substantial inter-annotator agreement (Cohen's κ = 0.72).
Dual-Aspect Evaluation Framework: 60 Complex Statutes (Penal, Civil, Land Law) × 4 Frontier LLMs × 2 Runs (480 outputs) → Aspect 1: Multi-Dimensional Benchmark (5-Rater Legal Accuracy, 253-User Lay Readability, Hybrid Automated/Human Consistency) + Aspect 2: 9-Category Expert-Validated Legal Reasoning Error Typology (κ = 0.72).
- · 60 Curated Vietnamese Legal Articles (Penal Code 2015, Civil Code 2015, Land Law 2024) · 480 Annotated Model Outputs
Weighted Legal Accuracy (1–5) · Layperson Readability (1–5, N=253 raters) · Composite Consistency (1–5) · 9-Category Error Frequency Matrix (Cohen's κ = 0.72)
Additional Figures from the Manuscript
Figure 2: Figure 2: Multi-dimensional trade-off between Legal Accuracy, Readability, and Consistency.
Primary Contributions
- 01Established the first dual-aspect evaluation benchmark of frontier LLMs on Vietnamese statutory simplification combining 253-participant readability ratings with expert legal evaluation.
- 02Developed a 9-category Legal Reasoning Error Typology validated with Hanoi Open University legal scholars (Cohen's κ = 0.72).
- 03Uncovered the 'Accuracy Illusion' and 'Alignment Tax' trade-offs, providing a risk-aware human-in-the-loop routing blueprint for legal AI.
Analyzed Boundary Conditions
- ·Evaluates zero-shot prompting across 60 purposively sampled statutes on closed-source frontier models; open-weight and RAG/CoT mitigations are explored in subsequent work (ViMELD).
BibTeX Entry
@inproceedings{le2026benchmarking,
author = {Le, Van-Truong},
title = {From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text},
booktitle = {Proceedings of the FISU Joint Conference on Artificial Intelligence (FJCAI 2026)},
year = {2026},
note = {arXiv:2604.16270}
}