Synchformer: Efficient Synchronization from Sparse Cues
Our objective is audio-visual synchronization with a focus on 'in-the-wild' videos, such as those on YouTube, where synchronization cues can be sparse. Our contributions include a novel audio-visual synchronization model, and training that decouples feature extraction from synchronization modelling through multi-modal segment-level contrastive pre-training. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale 'in-the-wild' dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability.
Code (2)
Tasks
Audio-Visual SynchronizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fast Sparsely Synchronized Brain Rhythms in A Scale-Free Neural Network
We consider a directed Barab\'{a}si-Albert scale-free network model with symmetric preferential attachment with the same in- and out-degrees, and study emergence of sparsely synchronized rhythms for a fixed attachment de…
RhythmSyncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers
Text-to-video and image-to-video generation have made rapid progress in visual quality, but they remain limited in controlling the precise timing of motion. In contrast, audio provides temporal cues aligned with video mo…
Video GenerationEffect of Inter-Modular Connection on Fast Sparse Synchronization in Clustered Small-World Neural Networks
We consider a clustered network with small-world sub-networks of inhibitory fast spiking interneurons, and investigate the effect of inter-modular connection on emergence of fast sparsely synchronized rhythms by varying …
CasCalib: Cascaded Calibration for Motion Capture from Sparse Unsynchronized Cameras
It is now possible to estimate 3D human pose from monocular images with off-the-shelf 3D pose estimators. However, many practical applications require fine-grained absolute pose information for which multi-view cues and …
Camera CalibrationSparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication
In large-scale reinforcement learning (RL) systems with decoupled Trainer-Rollout execution, the Trainer must regularly synchronize policy weights to the Rollout side to limit policy staleness. When inter-node bandwidth …
Reinforcement Learning