EE-USAD: Moving Beyond Objects to Understand How Data Makes Us Feel

TL;DR

The EE-USAD workshop (ACM Multimedia 2018) marks a pivotal transition in the AI community: moving from identifying what is in an image to understanding how that image makes a human feel. By focusing on Evoked Emotions (EE), the workshop bridges the gap between signal processing and social psychology, introducing novel concepts like Artificial Empathic Memory and depth-aware subjective colorization.

Background: The Subjective Frontier

For decades, the "Gold Standard" in computer vision was objective accuracy—identifying a dog as a dog or a chair as a chair. However, as social media has become ubiquitous, the subjective attributes of data (e.g., virality, beauty, interestingness) have become more valuable than literal content. The difficulty lies in the fact that these attributes are not inherent to the pixels but emerge from the interaction between the data and the human observer.

The Core Challenge: Why Subjectivity is Hard

The organizers identify five fundamental roadblocks to mastering Evoked Emotions:

1. Annotation Sparsity: Collecting subjective labels is more expensive and noisier than binary classifications.
2. Representation Gap: Standard CNN features might capture textures but miss "mood."
3. Evaluation Reliability: How do you "prove" an emotion detector is correct when human observers disagree?
4. Cultural Specificity: A scene that evokes "peace" in one culture might evoke "isolation" in another.
5. Psychological Integration: Theoretical models from psychology are rarely translated into computational code.

Methodology & Key Contributions

The workshop presented four breakthrough papers that address different facets of subjectivity:

1. Artificial Empathic Memory (AEM)

Current AI lacks "empathy" because it doesn't understand episodic memories—the personal history a user brings to an interaction. The proposed AEM architecture suggests that media technologies should maintain a psychological profile of a user's past emotional triggers to better anticipate future reactions.

2. DeepNSM: Predicting Memorability

Why do we remember some landscapes and forget others? Research here found that high-level scene categories are the strongest predictors of memorability. The authors introduced DeepNSM, a deep neural network that uses scene semantics to estimate how likely an image is to "stick" in a person’s mind.

3. Depth-Aware Colorization

One major "subjective" failure in AI is "color bleeding"—where colors spill across object boundaries. Note: Higher-quality depth information acts as a structural guide, ensuring that the subjective "beauty" of a colorized image is maintained by respecting physical boundaries.

4. Perceptual Similarity with Heat2Vec

Defining similarity for complex event sequences is notoriously hard. The authors created Heat2Vec, a CNN that looks at temporal heatmaps. By training on color, opacity, and position, the model achieved a ranking system that aligns almost perfectly with human perceptual judgment.

Critical Results & Comparison

The efficacy of these methods was validated through:

• Human-in-the-loop Studies: Heat2Vec was tested against 132 human participants, outperforming 11 different baselines.
• Objective/Subjective Hybrid Metrics: The colorization network was evaluated not just on PSNR (objective) but on user preference (subjective), showing that depth-guidance creates far more "natural" results.

Deep Insights & Future Outlook

The EE-USAD workshop proves that the "Subjective Gap" is closing. The real takeaway is that Multi-modality is the only way forward. To understand emotion, a model cannot look at pixels in isolation; it must consider depth (spatial context), category (semantic context), and even the user's episodic memory (temporal context).

Limitations: The current work still struggles with the "long-tail" of cultural differences. Most models are still trained on Western-centric datasets, which may bias the "Evoked Emotion" predictions.

Conclusion: This research trajectory is essential for the future of entertainment, digital therapy, and social media. As AI moves from being a "tool" to being a "companion," understanding our subjective attributes is no longer optional—it is the core requirement.