VP-VAE: Rethinking Vector Quantization via Adaptive Vector Perturbation

2026-02-19 · Linwei Zhai, Han Ding, Mingzhi Lin, Cui Zhao, Fei Wang, Ge Wang, Wang Zhi, Wei Xi arxiv

Vector Quantized Variational Autoencoders (VQ-VAEs) are fundamental to modern generative modeling, yet they often suffer from training instability and "codebook collapse" due to the inherent coupling of representation learning and discrete codebook optimization. In this paper, we propose VP-VAE (Vector Perturbation VAE), a novel paradigm that decouples representation learning from discretization by eliminating the need for an explicit codebook during training. Our key insight is that, from the neural network's viewpoint, performing quantization primarily manifests as injecting a structured perturbation in latent space. Accordingly, VP-VAE replaces the non-differentiable quantizer with distribution-consistent and scale-adaptive latent perturbations generated via Metropolis--Hastings sampling. This design enables stable training without a codebook while making the model robust to inference-time quantization error. Moreover, under the assumption of approximately uniform latent variables, we derive FSP (Finite Scalar Perturbation), a lightweight variant of VP-VAE that provides a unified theoretical explanation and a practical improvement for FSQ-style fixed quantizers. Extensive experiments on image and audio benchmarks demonstrate that VP-VAE and FSP improve reconstruction fidelity and achieve substantially more balanced token usage, while avoiding the instability inherent to coupled codebook training.

📄 PDF Abstract BibTeX arXiv:2602.17133

Code (0)

등록된 구현이 없습니다.

Tasks

Representation Learning

Beyond Stationarity: Rethinking Codebook Collapse in Vector Quantization

2026-02-21 · Hao Lu, Onur C. Koyun, Yongxin Guo, Zhengjie Zhu 외 arxiv

Vector Quantization (VQ) underpins many modern generative frameworks such as VQ-VAE, VQ-GAN, and latent diffusion models. Yet, it suffers from the persistent problem of codebook collapse, where a large fraction of code v…

Communication-Efficient Split Learning via Adaptive Feature-Wise Compression

2023-07-20 · Yongjeong Oh, Jaeho Lee, Christopher G. Brinton, Yo-Seb Jeon

This paper proposes a novel communication-efficient split learning (SL) framework, named SplitFC, which reduces the communication overhead required for transmitting intermediate feature and gradient vectors during the SL…

Quantization

Optimal and Near-Optimal Adaptive Vector Quantization

2024-02-05 · Ran Ben-Basat, Yaniv Ben-Itzhak, Michael Mitzenmacher, Shay Vargaftik

Quantization is a fundamental optimization for many machine-learning use cases, including compressing gradients, model weights and activations, and datasets. The most accurate form of quantization is \emph{adaptive}, whe…

Quantization

Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization

2026-06-01 · Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou, Naveen Verma arxiv

Massive activation spikes in Large Language Models (LLMs) severely degrade quantization by stretching dynamic ranges. While prior hypotheses characterize these as high-level scalar biases, we argue that they are merely t…

VQ-Seg: Vector-Quantized Token Perturbation for Semi-Supervised Medical Image Segmentation

2026-01-15 · Sicheng Yang, Zhaohu Xing, Lei Zhu arxiv

Consistency learning with feature perturbation is a widely used strategy in semi-supervised medical image segmentation. However, many existing perturbation methods rely on dropout, and thus require a careful manual tunin…

Semi-supervised Medical Image Segmentation Image Reconstruction