A benchmark scored by practicing surgeons
The paper asks whether state-of-the-art video generators can produce plausible surgical footage, not just realistic-looking footage. To test this, the authors built SurgVeo, a benchmark of 50 surgical clips drawn from two public repositories: laparoscopic hysterectomy (18 clips, from AutoLaparo) and endoscopic pituitary surgery (32 clips, from PitVis). Each sample pairs a starting frame with a real 8-second continuation that serves as the reference for judging a generated video.
Two models were tested without any surgical fine-tuning: the commercial Veo-3 and the open-source Wan2.2. Given the starting frame and a text prompt, each model generated an 8-second continuation. A panel of four board-certified surgeons scored the outputs, with two gastrointestinal surgeons on the laparoscopic track and two neurosurgeons on the endoscopic track; each track paired one senior and one junior surgeon. In total the study collected 4,800 individual expert ratings.
The pyramid separates appearance from surgical reasoning
The scoring instrument is the Surgical Plausibility Pyramid, a four-tier framework that moves from surface appearance to clinical reasoning. At the base sits Visual Perceptual Plausibility, meaning clarity and stability of the image. Above it are Instrument Operation Plausibility (whether instrument manipulation is accurate and technically sound), Environment Feedback Plausibility (whether the scene responds credibly, for example how tissue and fluids react), and at the apex Surgical Intent Plausibility (whether the action matches what the procedure calls for).
Surgeons scored each generated clip on a 1-to-5 integer scale at three cumulative time marks: 0–1 s, 0–3 s, and 0–8 s. Because each mark covers everything generated up to that point, the three scores show how plausibility holds up or decays as the model predicts further ahead. This design is what lets the study separate a visual rendering problem from a surgical reasoning problem.
Visual scores stay high while logic scores collapse
The central result is a split between the bottom and top of the pyramid. Visual Perceptual Plausibility stayed relatively high and stable across the 1, 3, and 8-second marks for both models and both procedures. For Veo-3 with the baseline prompt, the initial visual score was 3.72 in laparoscopic hysterectomy and 3.88 in endoscopic pituitary surgery; Wan2.2 reached 3.97 and 3.89 at 1 second on the same tracks.
The three higher dimensions behaved differently. In the Veo-3 laparoscopic baseline condition, Environment Feedback Plausibility fell from 3.06 at 1 second to 1.64 at 8 seconds, a drop of nearly 46%, and Surgical Intent Plausibility fell from 3.11 to 1.61. In endoscopic pituitary surgery, Veo-3 Surgical Intent dropped to 1.12 at 8 seconds with the baseline prompt. Wan2.2 showed the same pattern: on the endoscopic baseline track, Surgical Intent fell from 3.17 at 1 second to 1.42 at 8 seconds. The authors describe this divergence between high visual scores and low logic scores as the plausibility gap.
Most errors are surgical logic failures, not rendering flaws
The error tally makes the gap concrete. Reviewing Veo-3 outputs across the whole benchmark, the surgeons attributed only 6.2% of failures in laparoscopic hysterectomy and 2.8% in endoscopic pituitary surgery to basic visual quality distortions. More than 93% of errors in both procedures were failures of surgical logic.
The four largest categories were surgical intent errors (21.9% laparoscopic, 22.0% endoscopic), environment feedback errors (17.2% and 22.0%), surgical instrument errors (17.2% and 17.7%), and inappropriate surgical operations (15.6% and 21.3%). In endoscopic pituitary surgery these four high-level categories alone accounted for 83% of all errors. The qualitative examples match the numbers: a fabricated instrument where the real procedure uses a standard scalpel, a dissector moved right when the procedure requires a leftward move, a suction tool pulling an entire block of gel-foam as if it were a solid attached mass, and a predicted intent of wiping the dura when the real action was injecting biologic glue.
Telling the model the surgical stage does not close the gap
One obvious explanation is missing context, so the authors compared a baseline prompt naming only the procedure type with a stage-aware prompt that explicitly stated the current surgical stage. If context were the bottleneck, the stage-aware prompt should have raised the higher-level scores. It did not do so consistently.
For Veo-3 in laparoscopic hysterectomy, Instrument Operation at 8 seconds was 1.78 with the baseline prompt and 1.69 with the stage-aware prompt, essentially unchanged. In endoscopic pituitary surgery, Surgical Intent at 8 seconds was 1.12 versus 1.17. Wan2.2 was mixed: stage-aware prompting improved Instrument Operation in the laparoscopic track from 1.81 to 2.00 at 8 seconds but lowered Environment Feedback in the endoscopic track from 1.45 to 1.11. The authors read this as evidence that the deficit is not only a lack of information but also a limited capacity to reason about domain-specific causal and procedural constraints.
Why visual training does not transfer to surgical causality
The paper's explanation is that these transformer-based video diffusion models are trained to optimize perceptual fidelity, which suits propagating texture, color, and local motion continuity. That objective does not enforce causal, physics-grounded, or procedurally constrained generation, so adding a more informative text prompt does not change what the model is optimized to do. The authors argue that scaling general-domain training may be insufficient because surgery follows an expert-sense of anatomy, physiology, and biomechanics rather than the everyday physics of natural scenes.
This is the authors' mechanistic interpretation rather than a directly measured result, and it shapes their proposed remedies: physics-informed constraints, domain-adapted reward models, or integrating surgical knowledge graphs at the generation level, alongside better prompting. The practical reading is that a model can look right frame by frame while still failing the test that matters clinically, namely whether the next action is the correct one.
What the evidence covers and where it stops
The evaluation covers two models in zero-shot configuration, two procedures, and 50 clips, with scores averaged across two evaluators per clip. The consistency of the gap across two surgical specialties and two architecturally different models is the authors' basis for calling it a structural challenge, but they explicitly frame the study as a pilot and call for systematic investigation across a broader panel of generative models. The prompt design space is also described as substantially underexplored.
The clinical conclusion the authors draw is direct: zero-shot generation is presently unsuitable for applications such as high-fidelity simulation for medical education, pre-operative rehearsal, or intra-operative guidance. The benchmark and pyramid are offered as the measurement foundation for closing that gap, not as evidence that it has been closed.
Where effort should go next
The paper's own priority ordering is the most useful takeaway. Because zero-shot generation is demonstrably insufficient, the authors put few-shot learning and task-specific fine-tuning first as the more tractable near-term path to procedurally coherent generation. Explicitly embedding surgical knowledge into world models comes second, and physics-informed modeling plus architectures designed to curb error accumulation over time comes third.
That ordering follows from the specific pattern in the data: visual scores held up while the three logic dimensions decayed with the prediction horizon, and extra stage context did not reliably help. For anyone judging a surgical video generator, the implication is to weight the higher pyramid dimensions and the 8-second endpoint over visual realism, since that is where the measured gap between convincing footage and correct surgery actually appears.
Sources used in this answer
Quantifying the plausibility gap in generative AI for surgical video generation with expert assessment
Expert surgeons found that Veo-3 and Wan2.2 produced visually convincing surgical videos but failed on surgical logic, with over 93% of errors being logic failures rather than visual quality deficits.
