GSD: Generalized Stochastic Decoding
Although substantial progress has been made in various text generation tasks, there remains a vast gap between current generations and human languages. One reason is that virtually all decoding methods currently developed are pragmatic to address the text degeneration problem, which exists in both deterministic and stochastic decoding algorithms. So, why text generated from these algorithms are divergent? What is the critical difference between these algorithms? Moreover, is it possible to design a generalized framework where existing decoding algorithms can be naturally connected, uniformly described, and mutually inspired? In this paper, we try to explore answers to these intriguing questions. Correctly, we propose a generalized decoding framework that can be used to describe and connect existing popular decoding algorithms. Based on the framework, we propose a novel implementation with a distinctive core from existing decoding algorithms. As far as we know, this is the first work trying to propose a generalized framework to bridge these decoding algorithms using formal theorems and concrete implementations. By setting up different conditions, our framework provides infinite space to develop new decoding algorithms. Experiments show that text produced by our method is closest to the characteristics of human languages. Source code and the generated text can be accessed from https://github.com/ginoailab/gsd.git.
Code (0)
등록된 구현이 없습니다.
Tasks
Text GenerationSimilar Papers 제목 키워드 기반
Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects
Autoregressive language models (ARMs) have been shown to memorize and occasionally reproduce training data verbatim, raising concerns about privacy and copyright liability. Diffusion language models (DLMs) have recently …
Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspective
Stochastic sampling strategies are widely adopted in large language models (LLMs) to balance output coherence and diversity. These heuristics are often inherited in Multimodal LLMs (MLLMs) without task-specific justifica…
Visual Question AnsweringMultimodal ReasoningDynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generation
Stochastic sampling strategies such as top-k and top-p have been widely used in dialogue generation task. However, as an open-domain chatting system, there will be two different conversation scenarios, i.e. chit-chat and…
Dialogue GenerationDiversityQuestion AnsweringRPNT: Robust Pre-trained Neural Transformer -- A Pathway for Generalized Motor Decoding
Brain motor decoding aims to interpret and translate neural activity into behaviors. Decoding models should generalize across variations, such as recordings from different brain sites, experimental sessions, behavior typ…
Self-Supervised LearningPost-FEC BER Benchmarking for Bit-Interleaved Coded Modulation with Probabilistic Shaping
Accurate performance benchmarking after forward error correction (FEC) decoding is essential for system design in optical fiber communications. Generalized mutual information (GMI) has been shown to be successful at benc…
Benchmarking