Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability
Latent diffusion models pair VAEs with diffusion backbones, and the structure of VAE latents strongly influences the difficulty of diffusion training. However, existing video VAEs typically focus on reconstruction fidelity, overlooking latent structure. We present a statistical analysis of video VAE latent spaces and identify two spectral properties essential for diffusion training: a spatio-temporal frequency spectrum biased toward low frequencies, and a channel-wise eigenspectrum dominated by a few modes. To induce these properties, we propose two lightweight, backbone-agnostic regularizers: Local Correlation Regularization and Latent Masked Reconstruction. Experiments show that our Spectral-Structured VAE (SSVAE) achieves a $3\times$ speedup in text-to-video generation convergence and a 10\% gain in video reward, outperforming strong open-source VAEs. The code is available at https://github.com/zai-org/SSVAE.
Code (0)
등록된 구현이 없습니다.
Tasks
Text-to-Video GenerationSimilar Papers 제목 키워드 기반
Delving into CLIP latent space for Video Anomaly Recognition
We tackle the complex problem of detecting and recognising anomalies in surveillance videos at the frame level, utilising only video-level supervision. We introduce the novel method AnomalyCLIP, the first to combine Larg…
Anomaly DetectionMultiple Instance LearningVideo Anomaly DetectionNeutralizing Gender Bias in Word Embeddings with Latent Disentanglement and Counterfactual Generation
Recent research demonstrates that word embeddings, trained on the human-generated corpus, have strong gender biases in embedding spaces, and these biases can result in the discriminative results from the various downstre…
counterfactualDisentanglementWord EmbeddingsNeutralizing Gender Bias in Word Embedding with Latent Disentanglement and Counterfactual Generation
Recent research demonstrates that word embeddings, trained on the human-generated corpus, have strong gender biases in embedding spaces, and these biases can result in the discriminative results from the various downstre…
counterfactualDisentanglementSentiment AnalysisWord EmbeddingsThe Malignant Tail: Spectral Segregation of Label Noise in Over-Parameterized Networks
While implicit regularization facilitates benign overfitting in low-noise regimes, recent theoretical work predicts a sharp phase transition to harmful overfitting as the noise-to-signal ratio increases. We experimentall…
Self-supervised debiasing using low rank regularization
Spurious correlations can cause strong biases in deep neural networks, impairing generalization ability. While most existing debiasing methods require full supervision on either spurious attributes or target labels, trai…
Self-Supervised Learning