Bridging the Gap: A Formal Ontology for Gesture-Driven Service-Oriented Architectures
Abstract-We present a system engineering approach for designing Service-oriented Architectures (SOA) for software applications that use gesture commands. The approach employs ontology for gesture-based interaction which was designed on three levels: user execution, system implementation, and gesture reflection. The ontology borrows concepts from several research communities interested in gestures such as human-computer interaction, pattern recognition, and cognitive psychology. We show how the ontology can be used in order to inform the design of Service-oriented Architectures for engineering new systems and applications and describe a software architecture design for controlling smart homes with gesture commands
The paper introduces a comprehensive gesture ontology designed to inform the engineering of Service-oriented Architectures (SOA). By synthesizing cross-disciplinary knowledge from HCI, pattern recognition, and cognitive psychology, the authors propose a three-level framework—Execution, Implementation, and Reflection—to standardize the development of gesture-controlled systems like smart homes.
TL;DR
Gestural interfaces are transitioning from niche gadgets to core components of smart environments. However, the lack of a standardized language between psychologists, hardware engineers, and software architects has slowed the adoption of robust gesture-based systems. This paper proposes a three-tier gesture ontology (Execution, Implementation, and Reflection) that serves as a blueprint for building modular, scalable Service-oriented Architectures (SOA) for gesture control, demonstrated via a smart home use case.
The "Tower of Babel" Problem in Gesture Research
For years, gesture research has suffered from disciplinary silos:
- HCI practitioners focus on user experience and feedback.
- Pattern recognition experts focus on Euclidean distances and Hidden Markov Models.
- Cognitive psychologists focus on the "why"—how gestures relate to human thought and discourse.
Without a unified ontology, developers of Service-oriented Computing (SOC) struggle to define "services" that are truly interoperable. A "swipe" gesture to one developer might be a "dynamic stroke" to another, leading to architectural friction when trying to integrate diverse hardware (like Kinect or Wii) into a unified smart home system.
Methodology: The Tri-Level Ontology
The authors break down the complexity of gestures into three logical planes:
1. The Execution Level (The "How")
This layer focuses on the physical production. It defines the Users (Novice vs. Expert), Body Movements (Arsm, Legs, Fingers), and Acquisition Devices (Wearables like data gloves vs. Environmental sensors like cameras).
- Key Insight: It introduces Execution Enhancers, recognizing that feedback (visual, audio, haptic) is a core part of the gesture execution loop, not just an afterthought.
2. The Implementation Level (The "What")
This is the computational heart. It distinguishes between Machine Readable Representations (Features, Encodings like InkML) and Human Readable Representations (Videos/Images for training).
- Architecture Integration: It maps recognition approaches (Statistical, Neural Networks) directly to specific software tools.
3. The Reflection Level (The "Why")
Borrowing from linguistics (Kendon’s continuum) and psychology (McNeill), this level classifies the meaning of gestures—categorizing them as Deictic (pointing), Iconic (describing objects), or Metaphoric (abstract concepts).
Practical Implementation: The Smart Home Case Study
The real power of this ontology is seen when it informs Service Identification. The authors demonstrate a system where a Wii Remote controls a smart home. By following the ontology, they derived five specific services:
- Acquisition Service: Handles low-level communication protocols.
- Representation Service: Constrains data into a standardized XML format.
- Management Service: Handles the database of training samples.
- Recognition Service: A modular block implementing Statistical classifiers (Nearest Neighbor).
- Distance-Computation Service: A reusable unit for calculating Dissimilarity (Dynamic Time Warping).
Critical Analysis & Conclusion
Why this matters
The shift toward SOA in gesture recognition allows for device agnosticism. By defining a "Gesture-Representation-Service," a system can theoretically swap a Wii Remote for a Leap Motion controller without rebuilding the entire recognition logic. This modularity is a prerequisite for the Semantic Web.
Limitations
While comprehensive, the ontology's "Reflection Level" relies on manual mapping of gestures to commands. In the era of modern AI, one might expect a more dynamic or "learned" mapping where the system autonomously understands the intent behind a "Natural Command" rather than requiring rigid pre-definition.
Takeaway
For developers building the next generation of smart environments, this paper provides a robust framework for organizing code not just around "detection," but around the entire lifecycle of a gesture—from the movement of a user's muscles to the execution of a web service.