Bridging the Affective Gap: Frequency-Domain Kernelized Features for Video Emotion Recognition

TL;DR

Recognizing emotions in user-generated videos is notoriously difficult due to the "affective gap"—the disconnect between pixel-level data and human sentiment—and the inherent noise of amateur recordings. This paper proposes a breakthrough by transforming CNN features using a polynomial kernel based on the Discrete Fourier Transform (DFT) and applying sparse representation denoising. The result is a more robust and discriminative video representation that sets new benchmarks on VideoEmotion-8 and Ekman-6.

The Problem: Why Spatial Features Aren't Enough

While Deep Convolutional Neural Networks (CNNs) have mastered object and action recognition, Emotion Recognition remains a challenge. There are two primary blockers:

1. High Intra-class Variation: "Joy" can look like a girl playing piano in a quiet room or fans screaming in a stadium. Spatial features often focus on the scene rather than the underlying sentiment.
2. The Noise Factor: Unlike professional movies, user-generated content (UGC) is plagued by camera shake, low resolution, and lighting issues. Standard models treat this noise as meaningful signal, leading to poor generalization.

Methodology: The Power of the Frequency Domain

The authors' central insight is that emotion information might be better captured in a transformed feature space rather than the time-domain sequence of frames.

1. From DFT to Polynomial Kernels

The Discrete Fourier Transform (DFT) is typically used for signal processing to move from time to frequency. The authors cleverly observe that the DFT equation can be viewed as a linear kernel. By extending this to a polynomial kernel, they can project frame-level features (from ResNet-152) into a high-dimensional space where emotional nuances are more separable.

Figure 1: Overview of the proposed approach involving feature extraction, space transformation, denoising, and pooling.

2. Denoising via Sparse Representation

To handle amateur video noise, the paper introduces a denoising step. Because different videos have different lengths, they use cubic interpolation to align signals to a fixed length ($L$). They then apply Locality-constrained Linear Coding (LLC) to reconstruct a "cleaner" version of the frequency signal based on a learned dictionary. This effectively "smoothes" the noise out of the temporal trajectory of the features.

Experiments: Breaking the SOTA

The researchers tested their approach on two major datasets: VideoEmotion-8 and Ekman-6.

Quantitative Success

The results confirm that each component adds value:

• Baseline (CNN only): 45.5% (VideoEmotion-8)
• With DFT: 47.2%
• With Polynomial Kernel: 48.3%
• With Denoising (Final): 49.7%

Notably, when this frequency-domain approach is combined with semantic attribute features like SentiBank, the performance jumps to 52.5%, outperforming complex multimodal fusion models that require much larger training sets.

Table 1: Recognition accuracy improvement through each stage of the proposed method.

The "Surprise" Effect

Interestingly, the model performed best on the emotion of "Surprise". This is attributed not just to a higher number of samples, but to the fact that surprise often involves distinct temporal changes that the kernelized frequency features are uniquely equipped to identify.

Figure 2: Confusion matrix on Ekman-6, highlighting the model's high precision for "Surprise".

Critical Insight & Conclusion

A significant takeaway from this paper is the comparison with Fine-tuning. The authors found that fine-tuning a ResNet on video frames didn't necessarily help, because individual frames are often "neutral" or "weak" representatives of a video's overall emotion. Instead, transforming the temporal relationship between frames via kernels proved much more effective.

While the method is powerful, it relies on a fixed interpolation length ($L$), which may lose some fine-grained temporal data in extremely long videos. However, for standard user-generated clips, this approach provides a robust, mathematically sound way to bridge the affective gap and suppress the noise of the real world.

Future Outlook: This kernelized feature methodology could easily extend beyond emotions to other temporal tasks like action recognition or anomaly detection in surveillance, where noise suppression and relationship modeling are equally critical.