Beyond Heatmaps: Bridging the Semantic Gap with Concept-Driven Explanations

TL;DR

While AI researchers love pixel-perfect heatmaps, human users often find them abstract and confusing. This paper from Cornell University investigates how to make Machine Learning (ML) explanations more "human" by grounding them in concepts (like "stripes" or "wings") rather than just pixels. By varying the granularity and contextual breadth of explanations, the study reveals a shocking truth: simplifying explanations too much can actually make people trust the AI less.

The "Interpretation Problem" in XAI

Most Explainable AI (XAI) today focuses on Feature Attribution. If a model classifies an image as a "Zebra," a saliency map highlights the pixels that mattered. But there is a mismatch: models reason with low-level data features, while humans reason with abstract concepts.

The authors identify two critical missing dimensions in current XAI:

1. Granularity: The level of detail. Is a heatmap of 100,000 pixels better than a simple circle around a "beak"?
2. Context: The "why me?" vs. "why them?". Does the model always look for "beaks" when identifying birds, or is this specific image an outlier?

Methodology: Crowdsourcing the "Human Mental Model"

To fix this, the researchers didn't just use math—they asked humans. They built a pipeline to collect what they call Concept-driven Explanations.

1. The Elicitation Pipeline

Crowdworkers were asked to use a lasso tool to highlight specific parts of an image (e.g., a tiger's paw) and tag them with natural language labels. These were aggregated to create a "Concept Corpus."

2. Implementation of Context and Granularity

The researchers then built prototypes that combined traditional Grad-CAM heatmaps with these human labels:

• Context was added via bar charts showing how often certain concepts (like "ears" or "fur") appeared in similar instances.
• Granularity was adjusted using SLIC segmentation to turn "noisy" heatmaps into "cleaner," region-based blocks of color.

Experimental Insights: More Is... More?

The team ran a "Predict the Prediction" task: users saw an explanation and had to guess how confident the AI was.

Key Findings:

• Context is King: Providing "Global Context" (showing how the model behaves across an entire class) significantly improved the users' ability to estimate model performance.
• The Granularity Paradox: Common wisdom suggests that "simpler is better" for non-experts. However, this study found that High Granularity (pixel-level) actually outperformed coarse, simplified regions. When explanations were too "abstract" (G3), user confidence plummeted.
• The "Hiding Something" Effect: In qualitative interviews, users viewing simplified explanations felt the model was "hiding something." The detail in the messy heatmaps gave users a sense of "control" and "transparency."

Critical Analysis & Conclusion

This work challenges the trend of "over-simplifying" AI. Instead of dumbing down the model's inner workings, we should be contextualizing them.

Takeaways for AI Architects:

1. Don't Fear the Heatmap: Users value the raw detail of feature attribution; it provides a "fidelity" that builds trust.
2. Bridge with Concepts: Use natural language labels to group these details into logical clusters (e.g., "Legs," "Eyes").
3. Show the Neighborhood: An explanation of a single image is a snapshot. To truly understand a model, users need to see the "Big Picture" (Global Context).

Future Outlook: As we move toward more complex models like Vision Transformers or Multimodal LLMs, grounding mathematical "attention" in human "concepts" will be the only way to keep AI-human collaboration transparent and safe.

Paper Citation: Swati Mishra and Jeffrey M Rzeszotarski. 2021. Crowdsourcing and Evaluating Concept-driven Explanations of Machine Learning Models. CSCW '21.