TS3-Codec: Transformer-Based Simple Streaming Single Codec

2024-11-27 · Haibin Wu, Naoyuki Kanda, Sefik Emre Eskimez, Jinyu Li

Neural audio codecs (NACs) have garnered significant attention as key technologies for audio compression as well as audio representation for speech language models. While mainstream NAC models are predominantly convolution-based, the performance of NACs with a purely transformer-based, and convolution-free architecture remains unexplored. This paper introduces TS3-Codec, a Transformer-Based Simple Streaming Single Codec. TS3-Codec consists of only a stack of transformer layers with a few linear layers, offering greater simplicity and expressiveness by fully eliminating convolution layers that require careful hyperparameter tuning and large computations. Under the streaming setup, the proposed TS3-Codec achieves comparable or superior performance compared to the codec with state-of-the-art convolution-based architecture while requiring only 12% of the computation and 77% of bitrate. Furthermore, it significantly outperforms the convolution-based codec when using similar computational resources.

📄 PDF Abstract BibTeX arXiv:2411.18803

Code (1)

ga642381/speech-trident 공식 구현 tf

Tasks

Audio Compression

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec

2026-03-06 · Junhyeok Lee, Xiluo He, Jihwan Lee, Helin Wang 외 arxiv

Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in…

MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation

2025-05-31 · Yakun Song, Jiawei Chen, Xiaobin Zhuang, Chenpeng Du 외

Neural audio codecs have made significant strides in efficiently mapping raw audio waveforms into discrete token representations, which are foundational for contemporary audio generative models. However, most existing co…

Language Modeling Language Modelling

SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec

2025-08-04 · Chunyu Qiang, Haoyu Wang, Cheng Gong, Tianrui Wang 외 arxiv

Speech codecs serve as a crucial bridge in unifying speech and text language models. Existing codec methods face several challenges in semantic encoding, such as residual paralinguistic information (e.g., timbre, emotion…

Contrastive Learning

CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference

2026-04-07 · Yulin Zou, Yan Chen, Wenyan Chen, JooYoung Park 외 arxiv

Video streaming analytics is a crucial workload for vision-language model serving, but the high cost of multimodal inference limits scalability. Prior systems reduce inference cost by exploiting temporal and spatial redu…

Spectrogram Patch Codec: A 2D Block-Quantized VQ-VAE and HiFi-GAN for Neural Speech Coding

2025-09-02 · Luis Felipe Chary, Miguel Arjona Ramirez arxiv

We present a neural speech codec that challenges the need for complex residual vector quantization (RVQ) stacks by introducing a simpler, single-stage quantization approach. Our method operates directly on the mel-spectr…