Beyond the Screen: Designing for the Flesh, Emotion, and Aesthetic Experience

TL;DR

In this influential work, Kristina Höök challenges the "efficiency-first" dogma of HCI. By analyzing systems like eMoto and Affective Diary, she argues that technology should not diagnose our emotions but empower us to experience and reflect on them. This "Third Wave" approach views the body and mind as an integrated whole, turning sensors into mirrors for our social and corporeal selves.

Problem: The Ghost in the Machine

For decades, Human-Computer Interaction (HCI) treated the human body as a biological peripheral—an input device for mice and keyboards. When "Affective Computing" emerged, it often fell into the trap of reductionism: attempting to "solve" human emotion by mapping face tracking or heart rates to static labels like "0.9 Happy."

Höök identifies the core missing link: Embodiment. We don't just "have" emotions; we enact them through our bodies, culture, and social interactions. If our tools ignore this "flesh," they remain instrumental toys rather than transformative experiences.

Methodology: The Interactional Approach

The paper advocates for an Interactional View of emotion, characterized by three core pillars:

1. Open Surfaces: Interfaces shouldn't tell you what they mean. Instead, they provide "ambiguous" feedback (colors, shapes, movements) that users can "inscribe" with their own meaning.
2. Affective Loops: A process where the user expresses an emotion physically (e.g., shaking a device), the system responds with a sensory output, and this output further influences the user’s internal state.
3. Non-Reductionism: Rejecting the idea that a sensor can "know" a user better than the user knows themselves.

Key Systems Analyzed

• eMoto: A mobile messaging tool where gestures (pressure, shaking) generate colorful, animated backgrounds. It treats SMS as a "body language."
• Affective Diary: A personal log that visualizes physical arousal (GSR) and movement as abstract characters on a timeline, prompting deep self-reflection.

Fig 1: eMoto samples showing how emotional arousal and valence are mapped to abstract expressive shapes.

Experiments and Lived Experience

The "results" in this domain aren't just millisecond improvements; they are shifts in human behavior.

• The "Agnes" Case: A user utilized "angry" sharp red shapes to express "intense love" to her boyfriend. This highlights why pre-defined labels fail—the "open surface" allowed her to communicate a specific, private intensity that a standard "heart icon" could not.
• Behavioral Change: In the Affective Diary study, a user named Erica discovered her physical stress persisted long after a meeting ended. This visual "evidence" allowed her to consciously adopt "calmer" behaviors, effectively using the system as a somaesthetic learning tool.

Fig 2: The Affective Diary visualization, where character posture and color represent movement and arousal.

Critical Insight: The Future of Learning

Höök’s most provocative claim is that Learning Technologies are currently too "abstract." They focus on the grammar and rules of knowledge while ignoring the "in situ" corporeal experience.

She envisions learning tools that:

• Use Tangible Interaction (like moving physical blocks to understand logistics).
• Leverage Location-Awareness (like the Savannah game where children play as lions).
• Prioritize Aesthetic Unity, treating the student as a "sensual, aesthetically creative being" rather than a data-processing unit.

Conclusion

This paper is a rallying cry to bridge the "dualism gap" between mind and body. By designing for "knowing, communicating, and experiencing through body and emotion," we stop building tools that merely serve us and start building tools that help us become more aware humans.

Takeaway: The next frontier of AI and HCI isn't just "smarter" systems, but "supple" ones that leave room for the messy, beautiful ambiguity of the human flesh.