Can locally hosted LLMs accurately answer clinical questions from Finnish EHRs?

A retrieval task, not a medical exam

The paper targets a narrower job than general medical question answering. Clinicians often need one specific fact buried in years of documentation — whether a tumor was on the left or right side, which receptor status was recorded, whether a treatment came before surgery. The authors call this clinical contextual question answering: the model receives a patient's record and a natural-language question, and must return the value documented in that record.

The test set came from 183 pseudonymized records in the Wellbeing Services County of Pirkanmaa, Finland, covering people with current or prior breast cancer, suspected breast cancer, or screening encounters without malignant findings. Each record was annotated with up to 21 predefined questions, and only clinically applicable patient-question combinations were kept, yielding 1,664 question-answer pairs. Answers were restricted to categorical labels or numeric values, with an "Unknown" option when information was absent or inconclusive.

Two features of the data matter for reading every result. The text was predominantly Finnish, with occasional English and Latin medical terminology and no translation or language normalization. And the inputs were data-minimized extracts stored in a 26-column table, not complete raw EHRs: the data-providing authority had already filtered them to breast-cancer-relevant content. Records ranged from 408 to 5,508 words, with a median of 1,792.

Fourteen models, no retrieval step, no network

Fourteen locally deployable, instruction-tuned open-source models from five families were benchmarked, spanning roughly 4B to 70B parameters. All inference ran inside a closed offline computing environment with no external data transmission, which is the paper's central privacy claim: patient text never leaves institutionally governed hardware.

The design deliberately skipped retrieval-augmented generation, the common pattern of first searching a record for relevant passages and then feeding only those passages to a model. Instead each model received the entire available extract, so the evaluation tests whether a long-context model can integrate information spread across temporally separated clinical events. That choice also explains why context length becomes a hard constraint: two Llama-Poro models could only be scored on a reduced subset of 955 pairs because longer records exceeded their input window.

Accuracy here means agreement with a single medical oncologist's annotations, not with a panel. That annotator relabeled the data after a two-month blinded interval and matched 96.8% of 1,740 clinically relevant instances; the discordant pairs were excluded rather than adjudicated, leaving a high-confidence but single-annotator reference standard.

The best models answer most questions correctly

Llama-3.1-70B led free-text generation at 95.3% accuracy. Qwen3-30B-A3B-2507, at roughly 30B parameters, reached 95.1%, a difference small enough that the authors treat the two as comparable. That matters because the smaller model also ran far cheaper: under the standardized Transformers configuration it produced responses in under 10 seconds across the evaluated records, while reasoning-oriented models such as DeepSeek-R1-Distill-Llama-70B needed up to 153 seconds per query.

Consistency was measured separately from accuracy. For Llama-3.1-70B, 97.3% of semantically equivalent question variants received identical responses, regardless of whether those responses were right. This is a distinct property: a model can be accurate on the primary wording and still change its answer when the same question is phrased differently, which is exactly the failure mode a clinician would never notice at the point of use.

Domain specialization did not pay off systematically. Medical-domain and Finnish-specialized models showed no consistent advantage over generalist models, and parameter count alone did not determine performance — the mid-sized Qwen model matched the largest models while using less memory.

Temporal and numeric questions are harder

Accuracy was not uniform across the 21 clinical questions. Treatment summaries such as tumor laterality or invasiveness, histological subtype, and receptor status were generally answered with high accuracy. Performance dropped for tasks requiring temporal reasoning over treatment timelines or interpretation of quantitative values.

The clearest example the authors give is determining whether a patient received neoadjuvant therapy, meaning treatment administered before surgery. Answering it requires integrating information from temporally separated events rather than reading a single field. Consistently, mean per-question accuracy was lower for numeric-output questions than for categorical-output questions, though accuracy varied substantially across individual questions.

One limit on explaining these gaps: the reference annotations did not record which source field supported each answer or include supporting evidence spans, so the authors could not test whether performance depended on where in the serialized record the relevant information sat. The per-question pattern is descriptive, not a demonstrated mechanism.

Quantization buys memory at an uneven accuracy cost

Running large models offline is a hardware problem as much as an accuracy problem. Under the standardized configuration, large models needed peak memory allocations exceeding 130 GB, while smaller models stayed below roughly 25 GB. Two models failed outright: Llama-4-Scout-17B-16E did not complete EHR inference, and GPT-OSS-20B handled shorter records but failed on longer inputs.

Low-precision quantization — storing model weights in fewer bits — was tested as the remedy. Four-bit precision largely preserved accuracy while cutting memory sharply. Llama-3.1-70B fell from 95.3% in bf16 to 92.1% at 4-bit while peak memory dropped from 148.2 GB to 53.6 GB; Qwen3-30B-A3B-2507 went from 95.1% to 94.1% while memory fell from 61.8 GB to 20.5 GB.

Eight-bit results were erratic and should not be read as a general property of 8-bit quantization. Llama-3.1-70B dropped to 65.1% and R1-Distill-Llama-70B collapsed to 11.2%, against 94.8% in bf16. The authors attribute these anomalies to interactions between specific models and the quantization implementation, and the practical takeaway they draw is that 4-bit is the more reliable memory-saving option in this setting.

High accuracy does not mean trustworthy confidence

In a separate constrained multiple-choice setting, each predefined option was scored by its conditional likelihood and the top option was selected, letting the authors read a confidence value off the model. Accuracy stayed close to the free-text results, but calibration — how well stated confidence tracks actual correctness — varied widely across architectures.

Llama-3.1-70B had the highest accuracy in this setting at 95.7% but a moderate expected calibration error of 0.123. Llama-3.3-70B was slightly less accurate at 95.0% with the lowest ECE at 0.039. Gemma-3-27B-it paired 94.1% accuracy with strong calibration (ECE 0.045), while MedGemma-27B-text-it had comparable accuracy but a maximum calibration error of 0.707. Smaller models generally showed larger calibration errors. A confident answer from a high-accuracy model is therefore not automatically a well-calibrated one.

The safety analysis is the paper's most consequential result. A medical oncologist reviewed all 78 of Llama-3.1-70B's free-text answers that the automated grader flagged as incorrect, blinded to the reference labels. Fifteen were reclassified as correct, 15 as clinically insignificant errors, and 48 as clinically significant — errors with potential to influence diagnostic reasoning, treatment decisions, or patient management. That is 2.9% of all 1,664 answers.

Rewording can turn a right answer into a harmful one

The second safety analysis isolates a risk that accuracy alone hides. The authors collected cases where the two semantically equivalent question versions produced inconsistent responses and one was correct while the other was not. Of 38 such pairs, expert review reclassified three as correct, leaving 35.

Within those, 16 pairs — 0.96% of all 1,664 primary questions — contained one correct answer and one clinically significant error. Another 19 pairs (1.14%) contained one correct answer and one clinically insignificant error. In other words, a clinician asking a reasonable rephrasing of the same question could receive a materially different and potentially harmful answer, with no signal in the output that anything changed.

This is why the paper treats consistency and calibration as adoption criteria alongside accuracy rather than as secondary metrics. A model that is right 95% of the time on one wording can still be unsafe if its errors cluster around particular phrasings and are delivered with unearned confidence.

A case for local deployment, not unsupervised use

The evidence supports a specific claim: locally hosted open-source models can retrieve predefined, patient-specific breast-oncology variables from longitudinal Finnish EHR-derived extracts without sending data outside the institution, and a 30B model can do so at accuracy comparable to 70B models with lower measured memory use and latency. The authors also note that their accuracy figures sit numerically within the range reported for related English-language evaluations, while cautioning that differences in clinical scope, source data, question design, and grading prevent direct comparison.

The boundaries are equally specific. The evaluation covered predefined breast-oncology variables, not unrestricted EHR question answering or open-ended instruction following. It used data-minimized extracts rather than complete records, and it did not test whether models can detect invalid question premises or non-applicable queries — a real deployment need the authors explicitly leave unevaluated. Measurements describe the standardized Transformers configuration and should not be read as production latency.

The practical priority that follows is a split between model selection and oversight. On this evidence, the choice between a 70B and a 30B model should weigh accuracy together with memory, latency, and quantization behavior, since the accuracy gap was small and the efficiency gap was not. But no configuration in the study earned unsupervised use: with 2.9% of answers carrying clinically significant errors and 0.96% of questions producing a correct answer under one wording and a significant error under an equivalent one, source attribution, prospective workflow validation, and human review remain conditions of deployment rather than optional refinements.

Sources used in this answer