VIB-Probe: Filtering the Internal Noise to Solve VLM Hallucinations

TL;DR

Vision-Language Models (VLMs) often "hallucinate" by generating text that contradicts visual input. While most solutions look at the final output, VIB-Probe looks inside. By applying Variational Information Bottleneck (VIB) theory to internal attention head outputs, this framework effectively filters out syntactic noise to identify and suppress the specific internal signals that lead to unfaithful generation. It sets a new SOTA in both detecting and mitigating hallucinations in a training-free, inference-time manner.

Problem: The Entanglement of Truth and Noise

Why do VLMs hallucinate? Recent interpretability research suggests that hallucinations are not random; they emerge progressively through the Transformer layers. However, looking at internal hidden states is like looking into a "high-dimensional storm." Truthful visual grounding signals are mixed with:

1. Linguistic Priors: The model's tendency to follow grammar or common phrases instead of the image.
2. Syntactic Noise: Task-irrelevant information used for sentence structure.

Previous probing methods (like RepProbing) treat all internal information equally, making them prone to overfitting on dataset-specific biases.

Methodology: The Information Bottleneck to the Rescue

The core insight of VIB-Probe is that we need to compress the internal representations to find the "minimal sufficient statistics" for hallucination.

1. Feature Extraction

Instead of looking at raw attention weights, VIB-Probe extracts the pre-projection attention head outputs ($O_{l,h}=A_{l,h}{V}_{l,h}$). This provides a comprehensive "snapshot" of how each head processes multimodal data across all $L$ layers and $H$ heads.

2. The VIB Detector

The framework uses an encoder $p_{h}eta(z|v)$ to map these high-dimensional tensors into a latent space $z$. The optimization objective is two-fold:

• Maximizing Prediction: Ensure $z$ is highly predictive of whether a token is a hallucination.
• Minimizing Information: Penalize the KL-divergence between the latent distribution and a simple Gaussian prior. This forces the model to discard "semantic nuisances" that aren't critical for detecting visual fidelity.

Figure: The three-stage pipeline: Feature extraction, IB encoding for detection, and gradient-based mitigation.

3. Gradient-based Mitigation

Once the probe is trained, we can "blame" specific parts of the model for a predicted hallucination. By computing the gradient of the hallucination risk $s_u$ with respect to each attention head, the authors identify hallucination-sensitive heads. During inference, if the risk is too high, the model dynamically suppresses these heads and regenerates the token.

Experimental Performance

VIB-Probe was tested against four base models, including LLaVA-1.5, LLaVA-1.6, and the recent Qwen2.5-VL.

Detection SOTA

The method shows significant gains in generative benchmarks (M-HalDetect and COCO-Caption), where hallucinations are more nuanced than simple "Yes/No" tasks.

Table: VIB-Probe consistently outperforms baselines like DHCP and MetaToken across diverse VLM architectures.

Transferability and Robustness

A key highlight is the generalization gap. When trained on simple datasets (POPE) and tested on complex ones (M-HalDetect), traditional probers fail (32.4% drop), while VIB-Probe remains stable. This confirms that the Information Bottleneck successfully captures "domain-invariant" hallucination signals rather than just memorizing the dataset.

Critical Analysis & Conclusion

Takeaway

VIB-Probe proves that we don't necessarily need more data or bigger models to fix hallucinations. Instead, we need better "mechanistic filters" to leverage the information already present in the model's intermediate layers.

Limitations

• White-box access: The method requires access to internal attention outputs, rendering it inapplicable to closed-source APIs like GPT-4o.
• Architectural Dependence: Currently optimized for Transformer-based architectures; its efficacy on newer architectures (like Mamba or SSM-based VLMs) is unexplored.

In conclusion, VIB-Probe represents a significant step forward in trustworthy AI by bridging the gap between interpretability (finding the heads) and control (suppressing them at runtime).