VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing
Recent advancements in diffusion models have significantly improved video generation and editing capabilities. However, multi-grained video editing, which encompasses class-level, instance-level, and part-level modifications, remains a formidable challenge. The major difficulties in multi-grained editing include semantic misalignment of text-to-region control and feature coupling within the diffusion model. To address these difficulties, we present VideoGrain, a zero-shot approach that modulates space-time (cross- and self-) attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt's attention to its corresponding spatial-disentangled region while minimizing interactions with irrelevant areas in cross-attention. Additionally, we improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention. Extensive experiments demonstrate our method achieves state-of-the-art performance in real-world scenarios. Our code, data, and demos are available at https://knightyxp.github.io/VideoGrain_project_page/
Code (0)
등록된 구현이 없습니다.
Tasks
Video EditingVideo GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Self-Modulating Attention in Continuous Time Space with Applications to Sequential Recommendation
User interests are usually dynamic in the real world, which poses both theoretical and practical challenges for learning accurate preferences from rich behavior data. Among existing user behavior modeling solutions, atte…
Dynamic Link PredictionSequential RecommendationVirtual Classification: Modulating Domain-Specific Knowledge for Multidomain Crowd Counting
Multidomain crowd counting aims to learn a general model for multiple diverse datasets. However, deep networks prefer modeling distributions of the dominant domains instead of all domains, which is known as domain bias. …
Crowd CountingMamba Modulation: On the Length Generalization of Mamba
The quadratic complexity of the attention mechanism in Transformer models has motivated the development of alternative architectures with sub-quadratic scaling, such as state-space models. Among these, Mamba has emerged …
Beamforming with Oversampled Time-Modulated Arrays
The time-modulated array (TMA) is a simple array architecture in which each antenna is connected via a multi-throw switch. The switch acts as a modulator switching state faster than the symbol rate. The phase shifting an…
V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models
Recent advances in causal interpretability have extended from language models to vision-language models (VLMs), seeking to reveal their internal mechanisms through input interventions. While textual interventions often t…