i-Treasures: Decoding the Human Soul through Multimodal AI and Sensorimotor Learning

TL;DR

The i-Treasures project represents a paradigm shift in cultural preservation. Instead of just filming a dancer or recording a singer, this system uses a suite of advanced sensors—from brainwave monitors to ultrasound tongue imaging—to "digitize" the underlying human skill. By applying semantic analysis and 3D serious games, it creates an interactive bridge between traditional masters and modern learners.

The "Living" Problem: Why Video is Not Enough

Intangible Cultural Heritage (ICH), such as rare singing styles or complex craftsmanship, lives only in the bodies of its practitioners. Most current efforts (like Europeana) focus on digitization—storing videos and photos.

However, video cannot teach the exact muscle tension of a potter's hand or the internal movement of a folk singer's tongue. As elders pass away, these "living treasures" disappear. The challenge is twofold:

1. Capturing the Unseen: How do we record internal physiological movements (vocal tract, brain activity)?
2. Effective Transmission: How do we provide feedback to a student who is trying to replicate a master’s performance?

Methodology: The Multisensory "Hyper-Helmet" and Data Fusion

The i-Treasures architecture is a masterclass in multimodal integration. The researchers didn't settle for off-the-shelf solutions; they built a custom Hyper-helmet equipped with an ultrasound probe, EGG sensors, and respiratory belts to capture the "invisible" mechanics of singing.

The Tech Stack:

• Acquisition: Kinect for body motion, Myo armbands for finger gestures, and EEG for emotional/brain states.
• Semantic Layer: They used Multi-Entity Bayesian Networks (MEBN) to handle the uncertainty of cultural expression. This allows the system to bridge the gap between "low-level" sensor data (X-Y-Z coordinates) and "high-level" concepts (a "Double Step" in a Tsamiko dance).
• Feedback Loop: A Pedagogical Planner translates these analytics into "Serious Games" where learners receive real-time scores comparing their movements to the master's skeleton.

Cracking the Cultural Code: Experimental Insights

One of the most impressive feats of the project was Stylistic Analysis. By extracting "Relational Features" (how far feet are apart) and "Effort Features" (weight distribution), the system could mathematically distinguish between the Greek Tsamiko dance and the Belgian Walloon dance.

Key Findings:

• Walloon dancers cross their feet significantly more than Tsamiko dancers.
• Tsamiko exhibits higher "feet apart sideways" values, indicating a wider stance.
• Innovation Score: Users rated the system's innovative approach at 4.18/5, highlighting its potential to make heritage attractive to younger, tech-savvy generations.

Table 1: Technical Assessment (Overall System Satisfaction)

Critical Perspective & Future Outlook

While i-Treasures is a breakthrough, it faces the "Usability vs. Accuracy" trade-off. The Hyper-helmet, while technologically superior, was noted by some practitioners to cause discomfort, potentially altering the very performance it sought to record.

The future of this work lies in sensor miniaturization. As wearable tech becomes less intrusive, the "i-Treasures" approach could move from labs to local community centers. Moreover, the Text-to-Song synthesis module suggests a future where AI can act as a permanent virtual "master," preserving the acoustic signature of endangered languages and singing styles for eternity.

Takeaway: Preservation is no longer about looking back at the past via a screen; it is about stepping into the master's shoes through data-driven sensorimotor immersion.