ECCV 2026

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Jingqi Tian1, Yiheng Du2, Haoji Zhang1, Yuji Wang1, Isaac Ning Lee1,
Xulong Bai1, Tianrui Zhu1, Jingxuan Niu1, Yansong Tang1✉
1Tsinghua Shenzhen International Graduate School, Tsinghua University, 2Peking University.
✉Corresponding authors.
ArXiv Code
DDAVS Introduction
(a) Query-based unidirectional alignment generates audio queries but only aligns audio to visual features in one direction.
(b) KNN-based bidirectional alignment retrieves audio semantics from a limited semantic space and applies bidirectional fusion across all layers.
(c) Our bank-grounded query-based delayed bidirectional alignment anchors audio queries in a complete semantic space and performs delayed bidirectional cross-attention only in later layers for more robust multimodal alignment.

TL;DR

We propose DDAVS, an audio-visual segmentation framework that disentangles audio semantics and performs delayed bidirectional modality alignment to robustly localize sounding objects at the pixel level. DDAVS introduces an Audio Query Module with a prototype memory bank, a contrastive optimization module, and a multi-stage Audio-Visual Alignment Module, achieving state-of-the-art performance on AVS-Objects and VPO benchmarks, especially in challenging multi-source, subtle, distant, and off-screen scenarios.

DDAVS Teaser
DDAVS consistently outperforms previous approaches in challenging scenarios.

Abstract

Audioā€“Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audioā€“visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audioā€“visual misalignment, DDAVS introduces dual cross-attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audioā€“visual segmentation conditions.

Pipeline

Pipeline
Overview of DDAVS framework.
(a) The Audio Query Module (AQM) encodes original and augmented waveforms into disentangled semantic queries anchored to a prototype memory bank.
(b) The Contrastive Optimization Module (COM) enhances query robustness through contrastive learning, used only during training.
(c) The Audio-Visual Alignment Module (AVAM) fuses audio queries with visual features via stacked alignment blocks, and a lightweight decoder outputs the sound-conditioned segmentation masks.

Results

Results
Results
Results

Ablation Studies

Component ablation
Ablation study
t-SNE visualization of audio representations
t-SNE visualization of audio representations.
(a) The baseline produces entangled clusters across categories, while DDAVS forms clearly disentangled source clusters.
(b) The baseline collapses mixture audio toward the dominant source, while DDAVS achieves smooth interpolation between components.
(c) The baseline exhibits cross-domain fragmentation, while DDAVS achieves consistent cross-domain alignment between AVSBench and ESC50.

Case Study

Qualitative results of DDAVS across the four challenging AVS scenarios.

Multi-class sources
Multi-class sources. DDAVS segments each distinct sounding category while suppressing silent objects.
Multi-instance sources
Multi-instance sources. DDAVS separates multiple instances of the same sounding class without merging them.
Small or distant sources
Small / distant sources. DDAVS localizes small or far-away sound sources that baselines tend to miss.
Off-screen sources
Off-screen sources. DDAVS suppresses spurious activations when the sound source is off-screen.

Citation

@article{tian2025ddavs,
  title={DDAVS: Disentangled Audio Semantics and Delayed Bidirectional Alignment for Audio-Visual Segmentation},
  author={Tian, Jingqi and Du, Yiheng and Zhang, Haoji and Wang, Yuji and Lee, Isaac Ning and Bai, Xulong and Zhu, Tianrui and Niu, Jingxuan and Tang, Yansong},
  journal={arXiv preprint arXiv:2512.20117},
  year={2025}
}