paper-with-me

홈 › Papers

Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models

2025-01-02 · CVPR 2025 1 · Jingfeng Yao, Xinggang Wang

Latent diffusion models with Transformer architectures excel at generating high-fidelity images. However, recent studies reveal an optimization dilemma in this two-stage design: while increasing the per-token feature dimension in visual tokenizers improves reconstruction quality, it requires substantially larger diffusion models and more training iterations to achieve comparable generation performance. Consequently, existing systems often settle for sub-optimal solutions, either producing visual artifacts due to information loss within tokenizers or failing to converge fully due to expensive computation costs. We argue that this dilemma stems from the inherent difficulty in learning unconstrained high-dimensional latent spaces. To address this, we propose aligning the latent space with pre-trained vision foundation models when training the visual tokenizers. Our proposed VA-VAE (Vision foundation model Aligned Variational AutoEncoder) significantly expands the reconstruction-generation frontier of latent diffusion models, enabling faster convergence of Diffusion Transformers (DiT) in high-dimensional latent spaces. To exploit the full potential of VA-VAE, we build an enhanced DiT baseline with improved training strategies and architecture designs, termed LightningDiT. The integrated system achieves state-of-the-art (SOTA) performance on ImageNet 256x256 generation with an FID score of 1.35 while demonstrating remarkable training efficiency by reaching an FID score of 2.11 in just 64 epochs--representing an over 21 times convergence speedup compared to the original DiT. Models and codes are available at: https://github.com/hustvl/LightningDiT.

📄 PDF Abstract BibTeX arXiv:2501.01423

Code (2)

hustvl/LightningDiT 공식 구현 pytorch
westlake-repl/leanvae pytorch

Tasks

Image Generation

Methods 이 논문이 사용한 방법론

Attention 설명 없음
BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…
Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…
Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…
Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…
Adam 설명 없음

Similar Papers 제목 키워드 기반

Denoising Vision Transformer Autoencoder with Spectral Self-Regularization

2025-11-16 · Xunzhi Xiang, Xingye Tian, Guiyu Zhang, Yabo Chen 외 arxiv

Variational autoencoders (VAEs) typically encode images into a compact latent space, reducing computational cost but introducing an optimization dilemma: a higher-dimensional latent space improves reconstruction fidelity…

F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation

2026-06-04 · Dinghao Zhou, Xingchen Song, Di Wu, Pengyu Cheng 외 arxiv

Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable. This mismat…

Taming Feed-forward Reconstruction Models as Latent Encoders for 3D Generative Models

2024-12-31 · Suttisak Wizadwongsa, Jinfan Zhou, Edward Li, Jeong Joon Park

Recent AI-based 3D content creation has largely evolved along two paths: feed-forward image-to-3D reconstruction approaches and 3D generative models trained with 2D or 3D supervision. In this work, we show that existing …

3D Generation3D ReconstructionComputational EfficiencyImage to 3D+1

Taming Sampling Perturbations with Variance Expansion Loss for Latent Diffusion Models

2026-03-22 · Qifan Li, Xingyu Zhou, Jinhua Zhang, Weiyi You 외 arxiv

Latent diffusion models have emerged as the dominant framework for high-fidelity and efficient image generation, owing to their ability to learn diffusion processes in compact latent spaces. However, while previous resea…

Image Generation

DiP: Taming Diffusion Models in Pixel Space

2025-11-24 · Zhennan Chen, Junwei Zhu, Xu Chen, Jiangning Zhang 외 arxiv

Diffusion models face a fundamental trade-off between generation quality and computational efficiency. Latent Diffusion Models (LDMs) offer an efficient solution but suffer from potential information loss and non-end-to-…

Computational Efficiency