MEC 2016: Pioneering Multimodal Emotion Recognition in the Chinese "Wild"

TL;DR

MEC 2016 marks the first major effort to benchmark multimodal emotion recognition specifically for the Chinese language. Using the CHEAVD database—comprising spontaneous clips from movies and TV—the challenge task sets a baseline for identifying eight emotional states in noisy, real-world environments. While the baseline Random Forest models achieved modest accuracy (around 24%), the challenge established a vital infrastructure for future Chinese affective computing.

Problem & Motivation: Beyond the Lab

Traditional emotion recognition research often relied on "acted" data recorded in silent, perfectly lit laboratories. This created models that failed miserably when faced with the "Wild"—occlusions, varying lighting, and background noise.

More importantly, emotions are culturally and linguistically nuanced. A model trained on English speakers may not capture the prosodic or facial subtleties of a Chinese speaker. The authors identified that despite the existence of challenges like AVEC or EmotiW, there was no dedicated benchmark for the Chinese naturalistic context.

Methodology: The Core Baseline

The MEC 2016 setup is divided into three tracks: Audio, Video, and Multimodal.

1. Data: CHEAVD

The database includes 2,382 segments from 238 speakers. Unlike previous datasets, it provides labels for 26 non-prototypical states, though the challenge focuses on the top 8 (Angry, Happy, Sad, Worried, Anxious, Surprise, Disgust, and Neutral).

2. Feature Extraction

• Acoustic: Uses the eGeMAPS (extended Genevan Minimalistic Acoustic Parameter Set) via the openSMILE toolkit. This covers 88 functional attributes including pitch (F0), jitter, shimmer, and MFCCs.
• Visual: Employs LBP-TOP (Local Binary Pattern - Three Orthogonal Planes). This is a powerful dynamic texture descriptor that analyzes the face across XY, XT, and YT planes to capture movement over time.

3. Architecture

The baseline utilizes Random Forests (100 trees). For multimodal fusion, the authors chose Feature-Level Fusion, which simply concatenates the audio and visual vectors into a single high-dimensional input for the classifier.

Experiments & Results

The results under "in-the-wild" conditions proved to be a reality check for the field.

• Class Imbalance: Initial models struggled with minority classes (e.g., Surprise, Disgust). The researchers had to implement SpreadSubsample (sampling) to ensure the training data was balanced, capping emotion classes at 100 instances.
• Performance: On the Test Set, multimodal fusion led to a Macro Average Precision of 30.63% when using balanced sampling.

Critical Analysis & Conclusion

Takeaway

MEC 2016 proves that multimodal fusion generally outperforms unimodal approaches in noisy environments, but the margin is slim when using simple feature concatenation. The "Wild" remains an open problem where traditional descriptors like LBP-TOP meet their limits.

Limitations

• Feature-Level Fusion: Simple concatenation ignores the temporal alignment and cross-modal correlations between audio and video.
• Sample Size: With only 2,852 clips, the dataset is relatively small for modern deep learning (though it was significant for 2016).

Future Outlook

The authors noted that participants using LSTM-RNNs (Long Short-Term Memory networks) achieved considerable improvements over the Random Forest baseline. This signaled the transition from handcrafted features/traditional ML to end-to-end deep temporal modeling in affective computing.