Regularized Vector Quantization for Tokenized Image Synthesis
Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. Predominant approaches learn the discrete representation either in a deterministic manner by selecting the best-matching token or in a stochastic manner by sampling from a predicted distribution. However, deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while stochastic quantization suffers from low codebook utilization and perturbed reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate above issues effectively by applying regularization from two perspectives. The first is a prior distribution regularization which measures the discrepancy between a prior token distribution and the predicted token distribution to avoid codebook collapse and low codebook utilization. The second is a stochastic mask regularization that introduces stochasticity during quantization to strike a good balance between inference stage misalignment and unperturbed reconstruction objective. In addition, we design a probabilistic contrastive loss which serves as a calibrated metric to further mitigate the perturbed reconstruction objective. Extensive experiments show that the proposed quantization framework outperforms prevailing vector quantization methods consistently across different generative models including auto-regressive models and diffusion models.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A multi-layer image representation using Regularized Residual Quantization: application to compression and denoising
A learning-based framework for representation of domain-specific images is proposed where joint compression and denoising can be done using a VQ-based multi-layer network. While it learns to compress the images from a tr…
DenoisingQuantizationControllable Skin Synthesis via Lesion-Focused Vector Autoregression Model
Skin images from real-world clinical practice are often limited, resulting in a shortage of training data for deep-learning models. While many studies have explored skin image synthesis, existing methods often generate l…
VAEVQ: Enhancing Discrete Visual Tokenization through Variational Modeling
Vector quantization (VQ) transforms continuous image features into discrete representations, providing compressed, tokenized inputs for generative models. However, VQ-based frameworks suffer from several issues, such as …
Relative Entropy Regularized Reinforcement Learning for Efficient Encrypted Policy Synthesis
We propose an efficient encrypted policy synthesis to develop privacy-preserving model-based reinforcement learning. We first demonstrate that the relative-entropy-regularized reinforcement learning framework offers a co…
Model-based Reinforcement LearningPrivacy PreservingQuantizationreinforcement-learning+1Vector quantization loss analysis in VQGANs: a single-GPU ablation study for image-to-image synthesis
This study performs an ablation analysis of Vector Quantized Generative Adversarial Networks (VQGANs), concentrating on image-to-image synthesis utilizing a single NVIDIA A100 GPU. The current work explores the nuanced e…
GPUImage GenerationQuantization