[Springer 2019] Beyond Simple Fusion: Multi-Level Context and Inter-Modal Attention for MSA

TL;DR

Multimodal Sentiment Analysis (MSA) is no longer just about merging features; it's about understanding context and relevance. This paper presents a sophisticated architecture that extracts context at unimodal, bimodal, and trimodal levels while using an attention mechanism to weigh the importance of inter-modal utterances. By moving beyond simple concatenation, the authors achieved a significant >3% accuracy boost over major baselines on standard benchmarks.

The Motivation: Why Simple Fusion Fails

In a typical video review, an "utterance" (the smallest unit of speech) rarely exists in a vacuum. Its sentiment is often colored by what was said before and after it.

Standard approaches typically use:

• Early Fusion: Concatenating features before processing.
• Late Fusion: Averaging individual modality predictions.

The authors argue these methods suffer from two major flaws:

1. Context Blindness: They ignore the sequential nature of dialogue.
2. Modality Inertia: They treat every modality as equally important, even if the visual signal is noisy or the audio is silent.

Figure 1: Traditional early vs. late fusion paradigms.

Methodology: Deep Contextual Hierarchy

The proposed methodology operates in a tiered structure to ensure no information is lost during the transition from raw data to sentiment labels.

1. Feature Extraction

The pipeline begins with extracting robust features:

• Audio: 6392 features via OpenSMILE (MFCCs, Pitch, etc.).
• Text: Word2Vec embeddings processed through CNN layers.
• Visual: 3D-CNNs to capture both spatial and temporal (motion) features.

2. Multi-Level Context Extraction

The core innovation lies in the Bidirectional Recurrent Neural Network (BiRNN) application. Unlike standard RNNs, BiRNNs look both forward and backward in the utterance sequence, effectively capturing the "flow" of a conversation.

3. Attention-Based Inter-Modal Fusion

Instead of forcing modalities together, the model uses Pairwise Attention Matrices. For instance, the system calculates how much the "Text" modality should influence the "Audio" modality for a specific utterance. This results in a dynamic representation where the most "informative" modality takes the lead.

Figure 2: The proposed multi-level context and attention fusion pipeline.

Experimental Results: Setting a New Baseline

The researchers tested their model on two cornerstone datasets: CMU-MOSI (Sentiment) and IEMOCAP (Emotion).

Key Findings:

• BiLSTM > GRU: Bidirectional Long Short-Term Memory units consistently outperformed Gated Recurrent Units, likely due to their superior ability to handle long-range dependencies in video segments.
• Trimodal Synergy: The combination of Text + Audio + Visual (T+A+V) reached the highest accuracy (80.87% on CMU-MOSI), proving that while text is a strong indicator, audio-visual cues provide vital nuance.
• Context Matters: Models including "Contextual Features" saw a sharp jump in performance compared to "Non-Contextual" versions.

Table 1: Comparison against SOTA baselines (Poria et al. and Zadeh et al.).

Final Insight

This work highlights a fundamental shift in Affective Computing: Relational context is as important as the data itself. By treating a video as a sequence of interconnected events rather than a bag of utterances, and by allowing modalities to "attend" to one another, the model mirrors the human way of perceiving emotions—where a sarcastic tone or a subtle facial micro-expression can completely change the meaning of a spoken word.

Limitations: The model struggles slightly in Audio+Visual (A+V) configurations without text, suggesting that textual data remains the "anchor" for high-accuracy sentiment analysis in current architectures.