[Affective Computing] Micro-Expressions vs. Heartbeat: Which Betrays Your True Emotional State?

TL;DR

Can a standard webcam read your mind? This study investigates the feasibility of recognizing abstract emotional states by analyzing two hidden "leaks": involuntary Micro-expressions and Pulse Rate Variability (PRV). By testing on the CAS(ME)^2 dataset, the researchers found that your heartbeat (captured remotely via skin color changes) is actually a more accurate teller of your mood than your facial expressions, achieving nearly 60% accuracy in a three-class classification task.

Background & Motivation: The "Hidden" Cues

In the realm of affective computing, macro-expressions (like a big smile or a visible frown) are the "low-hanging fruit." However, humans are experts at masking emotions. This brings us to two subtle indicators:

1. Micro-expressions: Involuntary facial flicks lasting less than 1/4 second.
2. Remote Photoplethysmography (rPPG): Measuring the rhythmic beating of the heart by detecting minute, blood-flow-induced color changes in the skin.

The authors argue that while facial expressions are discrete events, emotional states are continuous. Therefore, a modality that tracks physiological trends (like PRV) might be better suited for long-term state recognition than one that tracks instantaneous muscle movements.

Methodology: Spatio-Temporal Textures and Chrominance

The study employs a dual-stream approach to extract features from standard RGB videos:

1. Facial Expression Stream (LBP-TOP)

To capture the evolution of a micro-expression over time, the authors use LBP-TOP. Unlike standard LBP which looks at a single image, LBP-TOP looks at three orthogonal planes (XY, XT, and YT). This allows the model to "see" the motion of a facial muscle as a temporal texture.

Figure 1: LBP-TOP planes for facial expression feature extraction.

2. Physiological Stream (rPPG to PRV)

The PRV pipeline is more complex:

• Detection: Viola-Jones detects the face; a Kalman filter tracks it.
• Extraction: Using the CHROM (Chrominance-based) algorithm, the RGB traces are transformed into a pulse signal.
• Analysis: Features like SDPP (Standard Deviation of pulse-to-pulse intervals) and RMSSD are calculated to describe the autonomic nervous system's activity.

Figure 2: The framework for non-contact pulse rate variability (PRV) estimation.

Experimental Analysis: PRV Takes the Lead

The team used the CAS(ME)^2 dataset, categorizing emotional states based on the excitation videos used to induce emotions (Disgust, Anger, and Happiness). To ensure robustness, they utilized the Leave-One-Subject-Out (LOSO) protocol, which is much more challenging than standard k-fold cross-validation because it tests the model on a human it has never seen before.

Key Findings:

• The PRV Advantage: PRV achieved 59.79% accuracy, whereas M/M-FEs only achieved 42.74%.
• Temporal Mismatch: Micro-expressions are too "fast" and sparse. A 2-minute video might contain only a few micro-expressions, making a "majority vote" classification difficult.
• Contradictory Signals: Some subjects showed "Happiness" expressions while watching "Anger" videos—a discrepancy that physiological heart signals are less likely to fake.

Table 1: Performance comparison between Facial Expressions and PRV.

Critical Insight & Future Outlook

The most striking takeaway is that (PRV) can be an interesting feature to estimate emotional states even in datasets not originally designed for it.

However, the study has limitations:

1. Lack of Baseline: The dataset didn't include "Neutral" state videos, which are crucial for normalizing heart rate data.
2. Dataset Size: With only 62 videos across 22 participants, the statistical power is limited.

Future Work: The authors plan to move toward Multi-modal Fusion. By combining the instantaneous "spark" of a micro-expression with the steady "hum" of the heartbeat, AI could finally bridge the gap between perceived behavior and internal biological reality.