GAViD & CAGNet: Decoding Collective Emotions through MLLM-Enhanced Context
GAViD: A Large-Scale Multimodal Dataset for Context-Aware Group Affect Recognition from Videos
The paper introduces GAViD, a large-scale multimodal dataset for Group Affect Recognition (GAR) featuring 5,091 video clips with aligned audio and MLLM-generated context metadata. It also proposes CAGNet, a cross-attention gated network that achieves a state-of-the-art 63.20% accuracy in valence classification by effectively fusing visual, acoustic, and textual modalities.
TL;DR
Understanding the "vibe" of a group is exponentially harder than reading a single face. This paper introduces GAViD, the most comprehensive video dataset for Group Affect Recognition (GAR) to date, enriched with metadata generated by Video-ChatGPT. Alongside the data, the authors present CAGNet, a model that uses cross-attention and gated fusion to prove that knowing where a group is (context) is just as important as how they look.
Perception Gaps in Group Affect
Traditional Group Affect Recognition (GAR) faces an "identity crisis" in the wild. While individual emotion recognition is mature, collective affect is shaped by:
- Environmental Complexity: Poor lighting and occlusions make face-only detection (a staple of prior SOTA) fail.
- Contextual Ambiguity: Is a group shouting because they are angry or because they are at a concert? Without scene metadata (e.g., "protest" vs. "party"), models often flip a coin.
- Data Scarcity: Most existing datasets (like VGAF or GECV) offer only ternary valence (Positive/Negative/Neutral) but lack discrete emotion labels (Happy/Sad/Fear) and high-level scene descriptions.
Methodology: The Power of Three (V+A+C)
The authors bridge this gap with a tri-modal approach: Vision (V), Audio (A), and Context (C).
1. The GAViD Dataset
The GAViD dataset (5,091 clips) isn't just bigger; it's smarter. By using Video-ChatGPT, the authors extracted detailed narratives for each scene. Human annotators then audited these descriptions and added action cues (e.g., "smiling," "clapping").
2. CAGNet Architecture
Unlike models that simply concatenate features, CAGNet employs a sophisticated alignment and gating strategy:
- Modality-Specific Encoding: Uses DINOv2 (Vision), Wav2Vec 2.0 (Audio), and XLM-RoBERTa (Context).
- Cross-Modal Alignment: Three cross-attention blocks allow each modality to "query" others. For instance, if the video is blurry, the model can look to the audio or text for clarity.
- Gated Fusion: A Squeeze-and-Excitation (SE) gate dynamically weights the importance of each modality based on the input.
Experimental Results: Context is King
The results reveal a clear hierarchy in modality effectiveness. While trilateral fusion (V+A+C) is the winner, the combination of Visual + Context performed remarkably well, often outperforming Visual + Audio.
| Model | Modality | Test Acc | Test F1 |
|---|---|---|---|
| LLaVA-NeXT | V+A | 54.82% | 0.388 |
| CAGNet (Ours) | V+A+C | 63.20% | 0.614 |
Key Insight: Handling Subtle Emotions
The qualitative analysis highlights that for "hostile" or "cooperative" interactions that look visually similar, the textual context (e.g., "a scene in an office where the boss is scolding his employee") acts as the tie-breaker that corrects the prediction.
Critical Analysis & Future Outlook
The introduction of GAViD marks a shift from Global Visual Features to Semantic Understanding in affective computing.
Strengths:
- MLLM Integration: Successfully demonstrates how generative AI can be used to "upcycle" raw video into feature-rich datasets.
- Robustness: CAGNet's "Modality Dropout" training ensures the model remains functional even if audio or text is missing at inference.
Limitations:
- The dataset still shows a heavy imbalance toward "Happy" and "Neutral" states, a common hurdle in "in-the-wild" data collection.
- The context is currently static (a single description per clip); future work could explore dynamic, frame-level context updates.
Conclusion
GAViD and CAGNet prove that to understand how we feel together, AI needs to understand the story we are in. By fusing MLLM-generated metadata with audio-visual cues, this research sets a new benchmark for computational social systems and human-computer interaction.