Can diffusion models morph sound while preserving temporal structure?

Structure and timbre are separated into two guidance axes

Existing diffusion-based morphing approaches entangle temporal structure with timbral identity, so there is no mechanism to hold one fixed while transforming the other. Smorph's core move is to decompose guidance into two composable axes: temporal structure, handled by hierarchical two-condition guidance, and timbral identity, handled by weighted multi-prompt text composition.

The practical consequence is that a user can morph from brass to strings at a fixed pitch, or shift a drum sample toward an everyday texture while keeping its rhythmic dynamics. Structure is not a post-hoc correction; it is a conditioning signal that anchors the trajectory while the timbral term moves.

The timbral term is measured against the structure-conditioned prediction

The guided velocity field combines an unconditional prediction, a structure-only prediction, and one prediction per text prompt. The key detail is that the timbral term is computed relative to the structure-conditioned prediction rather than the unconditional baseline.

In the two-prompt case the per-prompt weights are set by a morph parameter that sweeps from one endpoint to the other; for more than two prompts the parameter generalizes to a simplex-valued set of weights. Each denoising step needs two plus N forward passes, which the authors note can be batched so the added computation time is minimal.

Three morphing modes share one guidance equation

Smorph supports prompt-to-prompt, audio-to-prompt, and audio-to-audio morphing. In prompt-to-prompt mode the structural control signal can be self-guided, where a source prompt generates a reference sound whose structure is then held, or externally-guided, where the user supplies a reference clip from which the structural signal is extracted.

In audio-to-prompt mode, the source audio plays two roles: its structural descriptors provide the control signal, and its encoded latent is forward-diffused to initialize denoising. The noise level at which denoising starts acts as a morphing parameter in its own right, with later starts preserving more source detail and earlier, noisier starts permitting more substantial transformation. In audio-to-audio mode, target audio is encoded and its global embedding replaces the text conditioning vector, keeping the guidance formulas unchanged.

Structure and source preservation improve over the baseline

Across NSynth, Clotho, and FSL-10K drum recordings, the evaluation uses 100 source-target pairs in within-domain settings (NSynth-NSynth at fixed pitch, Clotho-Clotho preserving RMS envelope) and two cross-domain settings (NSynth-Clotho, testing whether a single note's pitch persists as notes shift toward non-musical texture, and Drumset-Clotho, testing whether real drum samples' RMS dynamics persist as percussive timbres shift toward everyday textures). All samples are 4 seconds.

In prompt-to-prompt morphing with structural control, smorph variants outperform the CoA baseline, but the effects differ by dataset setting. The paper reports this as substantially improved temporal-structure and source preservation, with the caveat that some settings show more conservative target-ward transformation.

Partial-noise initialization preserves the source better than embedding-only conditioning

For audio-to-audio morphing, the paper compares two ways of introducing the source: as a latent initialization (SDEdit) or as an embedding-based condition (FLAM). The FLAM-to-FLAM variant sweeps the morph parameter using FLAM embeddings of both recordings, while the SDEdit-plus-FLAM variant sweeps the starting noise level over all 24 denoising steps using the source latent as initialization and the target FLAM embedding as the target condition.

The reported result is that partial denoising improves preservation over FLAM-only conditioning, while trajectory and structure metrics vary by setting. This is a within-paper comparison, not a claim that one variant dominates everywhere.

Musicians found the trajectories expressive and playable

In an exploratory user study, six practicing musicians (ages 26-37, with 2-28 years of production experience) explored a 2D text-defined morph canvas connected to a pitch-mapped keyboard in a 55-minute session, with endpoints limited to text-only. Participants built morph spaces by designating all four endpoints and varied control signals (RMS, pitch on or off).

Two themes emerged: making uncertainty playable through structured serendipity, and enabling continuous sonic palettes from discrete parameters. Participants described traversal as predictable in direction and smooth, with high creativity support ratings for exploration, enjoyment, satisfaction, and results-worth-effort. The authors interpret this as suggesting structural anchoring can serve as a productive constraint for instrumental interaction, while noting the study is exploratory and small.

Structure-preserving morphing remains an open challenge

The paper is explicit that with multiple conditioning signals one can dominate the other, and that a trajectory appearing smooth in embedding space may not feel perceptually linear or musically meaningful. The authors frame perceptual linearization and richer interaction mechanisms as future work.

The evaluation is also bounded: the objective comparisons use 100 source-target pairs across the listed datasets, the user study has six participants, and the reported gains come with more conservative target-ward transformation in some settings. These conditions qualify how far the preservation advantage should be read.

Choose the mode by what must stay fixed

The decision-relevant split in this paper is not which variant is best overall but which constraint matters for the task. If the priority is holding rhythmic timing or pitch contour while timbre changes, the structure-guided prompt-to-prompt mode with an explicit structural control signal is the relevant configuration. If the priority is small timbral edits to an existing recording, partial-noise initialization preserves the source better than embedding-only conditioning. If both endpoints are recordings, the audio-to-audio variants trade off structure and trajectory metrics differently by dataset setting.

The most useful takeaway is that structural anchoring is the variable that makes morph trajectories playable and source-preserving, and that the cost of that anchoring is a more conservative move toward the target. Readers comparing morphing systems should therefore check which structure signal is held, how the source is initialized, and how far the target is actually reached, rather than treating smoothness alone as the success criterion.

Sources used in this answer