Edge4Emotion: Bridging the Gap Between Human Emotions and Software Engineering via Edge Computing

TL;DR

Edge4Emotion is a novel platform that brings multi-modal emotion recognition (Video, Audio, Physiological data) to the front lines of Software Engineering. By leveraging Edge Computing, it overcomes the latency of the cloud and the inaccuracy of single-source data, providing software teams with real-time "emotional telemetry" during requirement workshops and usability trials.

Background: Positioned at the intersection of Affective Computing and Human-Centric Software Engineering (HCSE), this work transitions the field from subjective "gut feelings" to objective, data-driven emotional analysis.

The Missing Dimension in SE: Human Emotion

Software engineering is inherently a human endeavor. However, traditional HCSE often treats stakeholders as black boxes or relies on retrospective text analysis. Such methods fail because:

• Single-source data is incomplete: A participant might say "it's fine" (Audio) while showing frustration (Video).
• Environment Dynamicism: Lab-based setups don't work for on-field testing (e.g., mobile apps in a gym).
• Latency Kills Interaction: If a facilitator receives emotional feedback 30 seconds late via the cloud, the moment for intervention has passed.

Methodology: The Edge4Emotion Architecture

The authors propose a hierarchical structure to decentralize the "brain" of the emotion recognition system.

The Three-Layer Stack

1. End Device Layer: Collects raw streams from IoT devices (smartphones, wristbands, microphones).
2. Edge Server Layer: This is the engine room. It hosts specialized CNN models for video/facial features and SVM/CNN models for audio spectrograms.
3. Application Layer: Translates raw emotion labels (Anger, Happiness, etc.) into SE-specific insights.

Figure 1: The hierarchical edge architecture for low-latency processing.

The Logic of Multi-Source Fusion

The fundamental insight is that different modalities capture different emotional granularities. While audio is excellent for identifying dominant emotional shifts (e.g., a sudden shout of disgust), video excels at capturing micro-expressions (e.g., a flicker of fear or surprise) that occur during complex software interactions.

Experimental Validation

The authors evaluated the platform using two distinct setups: one focused on video processing and the other on audio analysis, followed by an integrated multi-source prototype.

Modality Comparison: Video vs. Audio

• Video Performance: As seen in Figure 2, the video-based CNN was able to segment subtle emotions like "fear" and "surprise" over the duration of the stream.
• Audio Performance: Conversely, audio analysis (Figure 3) tended to be more polarized, often grouping emotions into "Neutral" or "Disgust," failing to perceive the nuance of the user's experience.

Figure 2: Fine-grained emotion detection via video.

Figure 3: Coarse emotion detection via audio.

Real-Time Prototype

The integration of these sources into a unified dashboard (Figure 4) allows for a "Multi-source Real-time" view. While the current implementation achieves near-real-time for audio (due to file processing constraints), the framework provides a clear path toward fully synchronized dual-stream inference.

Figure 4: The Edge4Emotion prototype UI integrating facial and vocal markers.

Critical Analysis & Conclusion

Takeaway

Edge4Emotion proves that for software engineering tools to be truly human-centric, they must be context-aware and low-latency. Moving the emotional "inference engine" to the edge makes it possible to deploy these tools in real-world scenarios, from gymnasiums to industrial fields.

Limitations

• Modality Synchronization: The current version still lags slightly in audio-video temporal alignment.
• Privacy: Processing emotional data at the edge is safer than the cloud, but the paper does not deeply address the "privacy-by-design" requirements for sensitive physiological data.

Future Work

The authors aim to open-source the platform and refine the multi-source fusion algorithms to handle more diverse SE activities, such as collaborative coding or emotional monitoring in remote software teams.