MSC: Multi-Scale Spatio-Temporal Causal Attention for Autoregressive Video Diffusion

2024-12-13 · Xunnong Xu, Mengying Cao

Diffusion transformers enable flexible generative modeling for video. However, it is still technically challenging and computationally expensive to generate high-resolution videos with rich semantics and complex motion. Similar to languages, video data are also auto-regressive by nature, so it is counter-intuitive to use attention mechanism with bi-directional dependency in the model. Here we propose a Multi-Scale Causal (MSC) framework to address these problems. Specifically, we introduce multiple resolutions in the spatial dimension and high-low frequencies in the temporal dimension to realize efficient attention calculation. Furthermore, attention blocks on multiple scales are combined in a controlled way to allow causal conditioning on noisy image frames for diffusion training, based on the idea that noise destroys information at different rates on different resolutions. We theoretically show that our approach can greatly reduce the computational complexity and enhance the efficiency of training. The causal attention diffusion framework can also be used for auto-regressive long video generation, without violating the natural order of frame sequences.

📄 PDF Abstract BibTeX arXiv:2412.09828

Code (0)

등록된 구현이 없습니다.

Tasks

Video Generation

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Causal Probabilistic Spatio-temporal Fusion Transformers in Two-sided Ride-Hailing Markets

2021-01-01 · Shixiang Wan, Shikai Luo, Hongtu Zhu

Achieving accurate spatio-temporal predictions in large-scale systems is extremely valuable in many real-world applications, such as weather forecasts, retail forecasting, and urban traffic forecasting. So far, most exis…

Causal Inference

CASE-NET: Deep Spatio-Temporal Representation Learning via Causal Attention and Channel Recalibration for Multivariate Time Series Classification

2026-05-21 · Fan Zhang, Yating Cui, Hua Wang arxiv

Multivariate time series (MTS) classification is foundational to pervasive computing and financial analysis, yet existing multi-scale paradigms are often constrained by suboptimal representation fidelity. We identify two…

Time Series Classification Representation Learning

CaST: Causal Discovery via Spatio-Temporal Graphs in Disaster Tweets

2026-02-01 · Hieu Duong, Eugene Levin, Todd Gary, Long Nguyen arxiv

Understanding causality between real-world events from social media is essential for situational awareness, yet existing causal discovery methods often overlook the interplay between semantic, spatial, and temporal conte…

Semantic Similarity

STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction

2025-08-17 · Haolong Chen, Liang Zhang, Zhengyuan Xin, Guangxu Zhu arxiv

Recently, spatio-temporal time-series prediction has developed rapidly, yet existing deep learning methods struggle with learning complex long-term spatio-temporal dependencies efficiently. The long-term spatio-temporal …

Representation Learning Contrastive Learning

CASC: Causal Adversarial Subspace Clustering for Multivariate Spatiotemporal Data

2026-07-23 · Francis Ndikum Nji, Vandana Janeja, Jianwu Wang arxiv

Deep subspace clustering plays a critical role in applications involving multivariate spatiotemporal data, such as sea ice monitoring, disease spread analysis, and tracking neuro-degeneration over time. Despite recent ad…