Can open-source LLMs perform legal document review without labeled training data?

What legal NLI asks a model to do

The paper frames document review as a task where human experts spend considerable hours reading legal documents, a burden for organizations including government inspectorates whose administrative inspections focus on detecting regulatory non-compliance. Natural language inference (NLI) is proposed as one computer-assisted route: the model reviews a document against a regulation or hypothesis and judges the relationship between them.

The evaluation uses ContractNLI, described in the paper as an English-language legal NLI dataset, plus two NLI4Wills datasets in the legal wills domain. The paper's stated motivation for open-source models is threefold: they can run locally and therefore be trusted with sensitive documents, they can be used without extensive fine-tuning through zero-shot and few-shot learning, and their accessible inner workings allow them to convey predictive uncertainty. The confidentiality of inspections is given as the reason API-based proprietary models would pose a security risk, and the domain specificity of inspections is given as the reason supervised methods would require substantial labeling for each new domain.

Why zero-shot matters more than the score

The relevant comparison in this paper is not simply which model scores highest, but what can be achieved with no labeled training data at all. The authors evaluate several state-of-the-art open-weight quantized LLMs on ContractNLI and report that they do not meet the supervised results but reach reasonable accuracy, up to 81%, without any finetuning.

This distinction matters because the supervised baseline, Span NLI BERT, was trained for the task. The paper reports that the authors successfully reproduced this baseline in order to verify the original results and confirm it as a reliable baseline. The zero-shot generative models are therefore being compared against a model that had the labeled data the generative models were denied.

The paper's own contribution list separates these claims: reproducing Span NLI BERT, evaluating open-weight quantized LLMs on ContractNLI, assessing generalizability to different legal domains via NLI4Wills, and conducting temperature and logit-based stability analyses.

Gemma-4 26B leads but stays below supervision

Among the generative models tested, Gemma-4 26B performs best, reaching an accuracy of 81.2%. The paper states that it even outperforms the supervised model on one metric, while on accuracy it is not possible to beat the supervised model with zero-shot approaches. The discussion quantifies the gap: Gemma-4 26B scores 5.7% below the supervised model.

These two statements must be read together. The single-metric win does not overturn the accuracy ranking, and the paper does not claim that zero-shot models match supervised performance on accuracy. The headline number for the generative side is 81.2% accuracy on ContractNLI, against a supervised baseline that remains ahead on that metric.

The paper also reports that Qwen-3.6 35B performs well on both ContractNLI and the additional datasets in the legal wills domain, which is the paper's evidence for cross-domain behavior rather than a second accuracy headline.

Ten models, three families, one task

The study investigates ten zero-shot LLMs across three model families on the ContractNLI NLI task. All models were quantized via Ollama, a detail the limitations section flags directly: because the tested set is quantized in this way, the authors state the results may not be representative of all quantized models' performance on this task.

The paper's stated advantages of open-source LLMs for this use case are local execution for sensitive documents, usability without extensive fine-tuning, and accessible inner workings for conveying predictive uncertainty. The evaluation design follows from those advantages rather than from a general model-ranking goal.

The limitations also note that the study mostly uses a fixed configuration for variables such as reasoning modes, structures, and prompts, so the results are described as non-exhaustive and the authors do not make claims about their setup versus alternate configurations for the same models.

Temperature stability and unreliable logit uncertainty

The paper conducts temperature and logit-based stability analyses and reports two distinct findings. First, the models are robust against changes in temperature. Second, logits for most models do not reliably reflect uncertainty. These are separate claims about separate properties and should not be merged into a single statement about model reliability.

The uncertainty analysis is reported through Mann-Whitney U tests with effect sizes across labels, seeds, and three ways of computing the metrics: logits of the whole response, only the label logits, and the first token of the predicted label. The results vary substantially by model and by metric definition. For example, in the Gemma-4 26B tables, several comparisons are not significant, while for Gemma-4 E2B the label-logit and first-token tables show p < 0.01 across labels and seeds.

The practical reading is that a model being stable across temperature settings does not imply its logits can be used as a confidence signal. The paper treats these as separate properties, and the limitations add that the stochastic nature of these models means different seeds and temperatures may still produce small classification variations even when input and explanation stay the same.

Where these results transfer and where they do not

The paper's generalizability evidence comes from validating on the NLI4Wills dataset to assess transfer to different legal domains, and it reports that Qwen-3.6 35B performs well on both ContractNLI and the additional datasets in the legal wills domain. That is the extent of the cross-domain claim supported here: one benchmark plus two wills datasets, all in English.

The limitations bound the result further. The tested model set is quantized via Ollama and may not represent all quantized models. The configuration for reasoning modes, structures, and prompts is mostly fixed, so the results are non-exhaustive and no claim is made against alternate configurations. The authors also note that while their procedures and temperature settings prevent label predictions from being misaligned from generated explanations, the stochastic nature of the models means this remains a small possibility.

The paper's stated next steps are to bring the models to the inspectorates' work practice to evaluate them for other languages and subdomains, and to experiment with uncertainty quantification methods, including using reasoning capabilities to provide explanations for the generated label. Those are plans, not results.

The decision that actually follows from this paper

The paper's own conclusion is conditional, not absolute: zero-shot, open-source, generative LLMs are a viable alternative for real-world legal NLI when no supervised data is available. The condition is the whole point. Where labeled data exists and a supervised model can be trained, the paper's accuracy result says the supervised model remains ahead, with Gemma-4 26B 5.7% below it.

So the operative split is between settings with and without labeled data, not between open-source and proprietary models in general. Within the no-labeled-data setting, the paper's ranking is Gemma-4 26B first among generative models at 81.2% accuracy, with Qwen-3.6 35B notable for holding up across both ContractNLI and the legal wills datasets.

The most useful thing a reader can take from this paper is that the 81.2% figure is a no-training-data result measured on ContractNLI, and that the paper's own framing treats it as a fallback for the no-supervision case rather than a replacement for supervised legal NLI. The temperature robustness finding should not be read as a confidence-calibration finding, since the same paper reports that logits for most models do not reliably reflect uncertainty.

Sources used in this answer