One framework plans and edits spatial audio scenes
Spatial Audio Generation and Editing with Executable Plans and Self-Distilling Transfusion · arXiv:2610.34415
In plain words
Spatial audio describes sounds coming from different places and moving over time. Editing it means changing a requested sound while keeping the rest of the scene intact.
The paper introduces Ambit, which writes a plan for every sound source and uses that plan to make four-channel spatial audio. In tested edits, it scored better than SwanWeave on six measures of content and sound reconstruction.
Separate measures found stronger preservation for unchanged sources, but only in intervals where those sources could be evaluated on their own.
The research
When an editor changes one sound in a spatial scene, can the system keep the other sources and their acoustic detail intact? The research question is how to connect decisions about each source with the audio rendering needed to make that kind of edit.
Ambit links full-scene source plans to four-channel audio rendering, then trains planning and rendering together on the student’s own editing contexts.
Ambit connects a full-scene source plan to four-channel audio rendering, with source audio supplying acoustic detail for edits. To test editing, the authors compare systems on the same requests using each system’s native interface. Ambit outperforms SwanWeave on all six reported content and spectral reconstruction measures.
Separate spatial measures show Ambit leading the target-scene, unchanged-source preservation, and object-direction groups. This is evidence across distinct aspects of editing; preservation is measured only in eligible intervals where an unchanged source is the sole annotated active source.
The current ScenePlan configuration supports scenes of about 15 seconds or less, and the paper does not evaluate whether acoustic details remain consistent across successive edits.
What the evidence shows
- The editing challenge
Spatial audio systems must coordinate each sound source’s content, active time, and movement. Editing adds a harder requirement: change selected attributes while keeping other sources and their acoustic detail.
Ambit is a framework for generating and editing four-channel spatial audio that represents sound in all directions. It addresses both tasks by connecting a description of the sound sources to audio rendering.
- Plans guide rendering
Ambit’s structured ScenePlan describes the full target scene, including sources that should remain unchanged. A compiler turns that plan into controls for an audio renderer, while source audio provides additional acoustic detail during editing.
The framework also trains its audio representation to reconstruct an omnidirectional channel and preserve relationships among channels. These design choices connect source-level decisions with the sound that is rendered.
- Spatial control results
The paper tests spatial generation by comparing Ambit with systems that use other planners or rendering pipelines. In the comparison that shares Ambit’s renderer, Ambit leads the reported scene-level and object-level spatial metric groups.
Scene measures compare directions in mixed audio; object measures use intervals with one annotated active source. The shared renderer makes this comparison evidence about planning’s role in this setup, while leaving other system comparisons broader.
- Editing and training evidence
For editing, Ambit outperforms SwanWeave on six content and spectral reconstruction measures, and leads the reported target-scene, unchanged-source preservation, and object-direction groups. Preservation applies only to eligible intervals where an unchanged source is the sole annotated active source.
The paper also finds that joint OPSD ranks best among tested post-training recipes and supervision variants on nine editing measures, supporting joint training in these evaluations.