[CVPR 2026] Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation

TL;DR

Researchers from UC Santa Cruz and Apple have introduced Kestrel, a potent training-free framework that effectively "debugs" Large Vision-Language Model (LVLM) hallucinations at inference time. By combining an external grounding agent (SAM3) with a conservative, multi-round verification process, Kestrel achieves leading results on POPE and MME-Hallucination benchmarks without needing a single parameter update.

The "Blind Trust" Problem in Multimodal AI

Despite the rapid scaling of models like Qwen and Llama, LVLMs still struggle with "seeing" what they are talking about. Most current mitigation strategies fall into two camps:

1. Decoding-based: Tweaking the internal probability distributions (e.g., OPERA, VCD).
2. Tool-based: Using a detector once and immediately trusting its output.

The authors argue that both are flawed. Decoding tweaks lack "hard" evidence, and one-pass tool usage often leads to over-correction, where the model flips a correct answer to an incorrect one because the tool was noisy.

Methodology: The Four Pillars of Kestrel

Kestrel transforms the black-box reasoning of LVLMs into a transparent, auditable pipeline.

1. Initialization and Claim Decomposition

Instead of verifying a whole paragraph, Kestrel breaks the answer into atomic, verifiable claims:

• “There is a cat” (Existence)
• “The cat is orange” (Color)
• “There are two cats” (Count)

2. Agent Grounding (The SAM3 Edge)

Kestrel utilizes SAM3, which allows for promptable concept segmentation. It produces three types of visual evidence:

• Segmentation Overlays: Precise pixel-level maps.
• Bounding Boxes: For spatial relationship calculations.
• Crop-and-Zoom Views: Enhanced local views for checking fine details like color or texture.

Figure 1: The Kestrel pipeline, showcasing the transition from claim decomposition to tool-based grounding and iterative refinement.

3. Claim-Level Verification

An "LVLM Judge" evaluates the claims against the collected evidence. Crucially, it must cite the specific evidence (e.g., "See Evidence ID: Box_1") it uses to support or contradict a claim.

4. Evidence-Gated Self-Refinement

This is the "brain" of the operation. Kestrel is conservative. It only changes its mind if the evidence is overwhelmingly strong and the confidence score is high (typically >0.82-0.90). This prevents the "hallucination of the tool" from corrupting the final output.

Experimental Showdown

Kestrel’s performance is notable because it scales across different backbones (Qwen3-VL, InternVL3.5) and consistently outperforms previous SOTA training-free methods.

• POPE Benchmark: Kestrel improved accuracy by +3.31% on average.
• MME-Hallucination: A massive leap of +28.34 points for Qwen3-VL, specifically excelling in position and count categories where previous models failed miserably.

Figure 2: Performance comparison on MME-Hallucination. Note Kestrel's dominance in object-level and attribute-level scores.

Why It Matters: Interpretability and Reliability

The most compelling aspect of Kestrel is its auditable evidence chain. In a human study, Kestrel was preferred in 68.3% of cases compared to other methods. Because the model provides a "paper trail" (verification traces), users can see exactly why a model changed its answer.

Limitations & Future Work

The primary cost of Kestrel is latency. By running multiple rounds of tool-calling and verification, it is roughly 15x-24x slower than a single-pass inference. Future research will likely focus on adaptive grounding—only calling tools when the model's internal uncertainty is high—to bridge the gap between accuracy and speed.

Conclusion

Kestrel represents a shift from "guessing" to "verifying" in the world of LVLMs. By grounding refinement in explicit visual evidence and maintaining a conservative update logic, it provides a blueprint for making multimodal AI trustworthy enough for real-world deployment.