paper-with-me

홈 › Papers

Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation

2023-07-25 · Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, Ya zhang

The goal of the audio-visual segmentation (AVS) task is to segment the sounding objects in the video frames using audio cues. However, current fusion-based methods have the performance limitations due to the small receptive field of convolution and inadequate fusion of audio-visual features. To overcome these issues, we propose a novel \textbf{Au}dio-aware query-enhanced \textbf{TR}ansformer (AuTR) to tackle the task. Unlike existing methods, our approach introduces a multimodal transformer architecture that enables deep fusion and aggregation of audio-visual features. Furthermore, we devise an audio-aware query-enhanced transformer decoder that explicitly helps the model focus on the segmentation of the pinpointed sounding objects based on audio signals, while disregarding silent yet salient objects. Experimental results show that our method outperforms previous methods and demonstrates better generalization ability in multi-sound and open-set scenarios.

📄 PDF Abstract BibTeX arXiv:2307.13236

Code (0)

등록된 구현이 없습니다.

Tasks

DecoderSegmentation

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…
Focus 설명 없음

Similar Papers 제목 키워드 기반

Revisiting Audio-Visual Segmentation with Vision-Centric Transformer

2025-01-01 · CVPR 2025 1 · Shaofei Huang, Rui Ling, Tianrui Hui, Hongyu Li 외

Audio-Visual Segmentation (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal. Prevailing AVS methods typically adopt an audio-centric Transformer architecture, where ob…

SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM

2025-11-18 · An Yu, Weiheng Lu, Jian Li, Zhenfei Zhang 외 arxiv

Video Moment Retrieval is a task in video understanding that aims to localize a specific temporal segment in an untrimmed video based on a natural language query. Despite recent progress in moment retrieval from videos u…

Moment Retrieval

TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis

2025-04-08 · Tri Ton, Ji Woo Hong, Chang D. Yoo

This paper introduces Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning (TARO), a novel framework for high-fidelity and temporally coherent video-to-audio synthesis. Built upon flow-based transform…

Audio SynthesisFAD

Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment

2023-07-24 · ICCV 2023 1 · Sarah Ibrahimi, Xiaohang Sun, Pichao Wang, Amanmeet Garg 외

Text-to-video retrieval systems have recently made significant progress by utilizing pre-trained models trained on large-scale image-text pairs. However, most of the latest methods primarily focus on the video modality w…

RetrievalText to Video RetrievalVideo AlignmentVideo Retrieval

Semantics-Aware Human Motion Generation from Audio Instructions

2025-05-29 · Zi-An Wang, Shihao Zou, Shiyao Yu, Mingyuan Zhang 외

Recent advances in interactive technologies have highlighted the prominence of audio signals for semantic encoding. This paper explores a new task, where audio signals are used as conditioning inputs to generate motions …

Motion Generation