Sync’n’Move: Turning Social Synchronization into a Musical Instrument

TL;DR

Sync’n’Move is an innovative mobile application that transforms social interaction into a creative tool. By measuring how "in-sync" two people move using their smartphone accelerometers, the system dynamically unlocks layers of a music track. Perfectly synchronized movements result in a full symphonic or pop arrangement, while disjointed movements leave the users with only a lonely metronome.

Background: Beyond Passive Listening

In the traditional media chain, users are often passive consumers. Even in social networks, the "social" aspect is frequently limited to likes and comments on static content. The SAME Project (Social Active Music Expression) aims to disrupt this by introducing active, embodied listening. The authors argue that the future of media lies in user-centric experiences where movements and social bonds directly shape the content.

The Problem: The Missing Social Link

Prior work in interactive music often focuses on the individual—one person's gestures controlling one filter. The challenge lies in quantifying the relationship between two people. How do we turn the subtle "dance" of two humans into a reliable control signal for a computer? Using simple thresholds for acceleration isn't enough; the system needs to understand the deep temporal dynamics of interpersonal coordination.

Methodology: The Math of Getting in Sync

The technical heart of Sync’n’Move lies in Phase Synchronization (PS) analysis. Instead of just looking at how fast someone shakes a phone, the system treats each user as a dynamical system.

1. Data Acquisition: Tri-axial accelerometers on mobile phones capture raw movement data, which is sent via OSC (Open Sound Control) to an EyesWeb XMI server.
2. Recurrence Plots (RP): The system maps these movements into a feature space. If a user’s state (movement pattern) repeats, it creates a "recurrence."
3. Correlation Probability of Recurrence (CPR): By comparing the recurrence patterns of two users, the system computes the CPR. A high CPR indicates that both users are "in phase"—essentially dancing to the same rhythm.

Figure 1: The architecture of the Sync’n’Move interaction loop.

The Interaction Logic:

• No Movement: Audio is inhibited.
• Low Sync / Single User: Only basic rhythmic elements (like a hi-hat or charleston) are played.
• High Sync: As the synchronization timer grows, more instruments are added (Bas, Guitar, Vocals), rewarding the duo with the full richness of the track.

Figure 2: The feature extraction and audio processing pipeline.

Experiments: Experts vs. Amateurs

The system was field-tested at the Agora Festival in Paris. The results provided a fascinating look at how different users perceive "control."

• Non-Experts: Loved the experience, giving high scores for "Fun" and "Understanding." For them, the magic of music appearing through movement was enough.
• Experts (Musicians): Were more critical. They noticed that while the presence of instruments was controlled by their sync, the tempo of the audio didn't always match their physical speed. This "temporal decoupling" led to lower scores for "Control" among professional users.

Figure 3: Global ratings showing high understanding but varied levels of pleasure and interest.

Critical Insight: The Future of Embodied Media

The genius of Sync’n’Move is not just in the signal processing, but in the Inductive Bias it imposes: it rewards social cohesion.

Limitations: As the evaluation showed, the mapping is currently "coarse." To satisfy expert users, future iterations must implement dynamic time-warping or beat-matching algorithms so the music's BPM follows the users' physical rhythm.

Conclusion: Sync’n’Move is a pioneering step toward a "User-Centric Future Internet." It proves that the "social dimension" isn't just about chatting—it's about moving together to create something beautiful.