Making Behavior Computable: Bridging Vision and Semantics via Knowledge Graphs

TL;DR

Analyzing human behavior in video is historically a labor-intensive manual task. This research presents a system that transforms raw video and deep learning detections into a structured RDF Knowledge Graph. By combining the precision of ontologies with the scale of Deep Neural Networks (DNNs), the authors enable experts to "query" behaviors (e.g., "Show me every time the caregiver supported the patient's elbow") as if they were searching a database.

Background & Motivation: The Gap in Video Understanding

While we have become adept at detecting "a person" or "a chair" in a video, understanding the contextual relationship and semantic meaning of their interaction remains a challenge.

• Prior Work Issues: End-to-end machine learning models are "black boxes" that lack explainability. They also require massive datasets that don't exist for niche domains like elderly care safety guidelines.
• The Insight: The authors argue that human-centric AI should incorporate "Internal Knowledge" (safety rules, anatomy, spatial relations). By using a formal Ontology, they create a bridge between the "what" (pixels) and the "why" (behavioral patterns).

Methodology: From Pixels to Knowledge Triples

The proposed framework follows a rigorous four-step pipeline:

1. The Multi-Layered Ontology

Instead of reinventing the wheel, the authors integrated several world-class ontologies:

• Anatomy (FMA): To define body parts like "Torso" or "Upper Limb."
• Spatio-temporal (Geosparql/Time): To handle where and when events occur.
• Behavior (DOLCE): To define actions (Actor/Operand/Result).

2. Multimodal Data Integration

The system doesn't rely solely on AI. It fuses:

• Manual Annotations: High-level context via ELAN.
• DNN Detectors: Low-level data from YOLO (objects) and 3DMPPE (3D pose estimation).

Fig 1: The holistic methodology integrating manual and automated data into a single semantic space.

3. Knowledge Graph Construction

By processing a single video of elderly care assistance, the system generated over 700,000 RDF triples. This turns a "flat" video file into a multi-dimensional graph where every frame is linked to anatomical parts, object identities, and action definitions.

Fig 2: An excerpt of the ontology showing how human body parts and actions are linked.

Evaluation: The "Transfer Assistance" Scenario

To test the system, the authors looked at a critical safety rule in elder care: "Place your body as close as possible to the receiver for stability."

An expert defined a rule: "Grab behavior where the actor (caregiver) holds the operand's (receiver) trunk using their upper limbs."

The system:

1. Converted this natural language rule into a SPARQL query.
2. Searched the Knowledge Graph.
3. Successfully returned the exact timestamps and frames where this condition was met.

Fig 3: The proof-of-concept web system showing the retrieved behavior locations within the video archive.

Critical Insight & Future Outlook

The true value of this work lies in Explainability. Unlike a standard AI model that might say "90% confidence of assistance," this system can explain precisely why a behavior was indexed (e.g., because the caregiver's upper limb was in contact with the patient's torso at time T).

Limitations: The study is currently a Proof of Concept (PoC) with a small dataset. The reliance on some manual entry (ELAN) suggests a bottleneck for massive-scale deployment.

Future Directions: The authors suggest using Graph Neural Networks (GNNs) and Graph Embeddings to automatically fill in "missing links" in the graph—effectively allowing the AI to "infer" actions even when the camera view is partially obstructed. This moves us closer to a world where AI doesn't just see video, but understands the human story within it.