Does Vector Quantization Fail in Spatio-Temporal Forecasting? Exploring a Differentiable Sparse Soft-Vector Quantization Approach

2023-12-06 · Chao Chen, Tian Zhou, Yanjun Zhao, Hui Liu, Liang Sun, Rong Jin

Spatio-temporal forecasting is crucial in various fields and requires a careful balance between identifying subtle patterns and filtering out noise. Vector quantization (VQ) appears well-suited for this purpose, as it quantizes input vectors into a set of codebook vectors or patterns. Although VQ has shown promise in various computer vision tasks, it surprisingly falls short in enhancing the accuracy of spatio-temporal forecasting. We attribute this to two main issues: inaccurate optimization due to non-differentiability and limited representation power in hard-VQ. To tackle these challenges, we introduce Differentiable Sparse Soft-Vector Quantization (SVQ), the first VQ method to enhance spatio-temporal forecasting. SVQ balances detail preservation with noise reduction, offering full differentiability and a solid foundation in sparse regression. Our approach employs a two-layer MLP and an extensive codebook to streamline the sparse regression process, significantly cutting computational costs while simplifying training and improving performance. Empirical studies on five spatio-temporal benchmark datasets show SVQ achieves state-of-the-art results, including a 7.9% improvement on the WeatherBench-S temperature dataset and an average mean absolute error reduction of 9.4% in video prediction benchmarks (Human3.6M, KTH, and KittiCaltech), along with a 17.3% enhancement in image quality (LPIPS). Code is publicly available at https://github.com/Pachark/SVQ-Forecasting.

📄 PDF Abstract BibTeX arXiv:2312.03406

Code (1)

Pachark/SVQ-Forecasting 공식 구현 pytorch

Tasks

AttributeComputational Efficiency Quantization regressionSpatio-Temporal ForecastingTraffic Prediction Transfer Learning Video Prediction Weather Forecasting

Methods 이 논문이 사용한 방법론

MAE 설명 없음

BASE 설명 없음

Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization

2026-04-16 · Umer Ahmed, Syed Ahmed Mahmood, Fawad Javed Fateh, M. Shaheer Luqman 외 arxiv

We propose a novel hierarchical spatiotemporal vector quantization framework for unsupervised skeleton-based temporal action segmentation. We first introduce a hierarchical approach, which includes two consecutive levels…

Action Segmentation

A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics

2026-04-16 · Fawad Javed Fateh, Ali Shah Ali, Murad Popattia, Usman Nizamani 외 arxiv

We present a novel hierarchical spatiotemporal action tokenizer for in-context imitation learning. We first propose a hierarchical approach, which consists of two successive levels of vector quantization. In particular, …

Vector Quantization in the Brain: Grid-like Codes in World Models

2025-10-16 · Xiangyuan Peng, Xingsi Dong, Si Wu arxiv

We propose Grid-like Code Quantization (GCQ), a brain-inspired method for compressing observation-action sequences into discrete representations using grid-like patterns in attractor dynamics. Unlike conventional vector …

MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion

2024-10-10 · Onkar Susladkar, Jishu Sen Gupta, Chirag Sehgal, Sparsh Mittal 외

The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key contributions to address the challenges of spatiotemporal video p…

Denoising parameter-efficient fine-tuning Quantization Text-to-Video Generation+3

Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidance

2024-03-18 · Hao Wu, Fan Xu, Yifan Duan, Ziwei Niu 외

This paper proposes a two-stage framework named ST-PAD for spatio-temporal fluid dynamics modeling in the field of earth sciences, aiming to achieve high-precision simulation and prediction of fluid dynamics through spat…

Quantization