Detecting Emotions: The Future of Automated UX Evaluation Through Machine Learning

TL;DR

Evaluating User Experience (UX) has traditionally been a manual, "high-touch" process. This paper presents an innovative leap toward fully automatic UX evaluation by predicting seven distinct emotions (Anger, Joy, etc.) using only mouse and keyboard interaction logs. By training models on data collected "in the wild," the authors bridge the gap between lab-based theory and real-world behavior.

Perspective: Why UX Evaluation is Stuck in the Lab

UX design is no longer just about "ease of use"; it's about the emotional resonance of a product. However, developers often skip rigorous UX testing because:

1. It is resource-demanding, requiring expert evaluators.
2. It lacks automation, relying on manual observations.
3. Existing data is often biased, as users in labs act differently than they do at home.

The authors identify a critical technical gap: the Peak-End Rule. Humans judge experiences based on the most intense point and the end, forgetting the frustrations in between. To solve this, we need a way to track emotions every second without interrupting the user.

Methodology: The "In-the-Wild" Dataset

The core contribution is a robust dataset built over 30 days. Unlike previous studies, this tracking was "transparent" yet "unrestricted," capturing interactions across 473 different websites.

1. The Data Capture Pipeline

• Interaction Logs: Mouse speed, acceleration, clicks, and keyboard rhythms.
• Ground Truth: Every 2 seconds, a webcam snapshot was analyzed by the Affdex SDK (facial expression analysis) to assign an emotion label (0-100).
• Time Windows: The researchers calculated features across 9 different windows (25ms, 50ms, 100ms) to see whether past or future behavior best predicts a current emotion.

The proposed dashboard uses heatmaps to visualize where users felt specific emotions (e.g., Joy) on a webpage.

2. Algorithmic Comparison

The study compared four heavyweights:

• Binary Decision Trees
• Random Forests
• AdaBoost
• Multi-Layered Perceptron (MLP)

Experimental Insights: What Drives Emotion?

The results revealed that Random Forests and Decision Trees are the most effective for this type of tabular interaction data.

Average accuracies across 7 emotions. Emotions like Surprise (57%) and Sadness (53%) were significantly easier to detect than Contempt.

Key Findings:

• The "After" Window is Best: The movement data immediately after a sampling moment was more predictive of the emotion captured at that moment, suggesting a slight lag between internal feeling and physical reaction.
• Emotion Specificity: Some emotions are "noisier" in interaction logs. While "Surprise" often correlates with specific mouse pauses or spikes, "Contempt" is much more subtle and harder for ML to distinguish from a baseline state.

Critical Analysis & Conclusion

This work provides a critical foundation for non-intrusive UX monitoring. However, there are limitations:

• Data Volume: Due to computational constraints, only 0.1% of the 3GB dataset was used for the final training.
• Sample Diversity: While "in the wild," the 12 participants are a small starting point.

The Takeaway

We are moving toward a world where a simple JavaScript snippet on a website can tell a developer exactly which menu item caused a user to feel "Disgust" or "Confusion." By bypassing self-reported surveys and using ML to interpret "digital body language," UX evaluation becomes continuous, objective, and scalable.

Future Outlook: The integration of Support Vector Machines (SVM) and increasing the training resolution to five or seven emotion classes (following the SAM questionnaire) will likely push accuracies even higher, making this tool a standard in the web developer's toolkit.