Error-Correcting Codes For Approximate Neural Sequence Prediction
We propose a novel neural sequence prediction method based on \textit{error-correcting codes} that avoids exact softmax normalization and allows for a tradeoff between speed and performance. Error-correcting codes represent predictions and targets as a binary code where each bit is represented by a logit. The codebook is arranged such that similar tokens are close to each other using word embedding similarity, ensuring that incorrect predictions are at least semantically close to the target. We also address the well-established problem of compounding errors by mixing the latent codes of past predictions and past targets in one of two ways: (1) according to a predefined sampling schedule or (2) a differentiable sampling procedure that replaces the argmax operation. Low dimensional codes show similar performance to models that use the full softmax and outperform alternative approximate methods for language modeling and text generation, while generation further benefits from our mixture sampling.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingPredictionText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Error-Correcting Neural Sequence Prediction
We propose a novel neural sequence prediction method based on \textit{error-correcting output codes} that avoids exact softmax normalization and allows for a tradeoff between speed and performance. Instead of minimizing …
Image CaptioningLanguage ModellingPredictionText GenerationDNA data storage, sequencing data-carrying DNA
DNA is a leading candidate as the next archival storage media due to its density, durability and sustainability. To read (and write) data DNA storage exploits technology that has been developed over decades to sequence n…
Model CompressionLLM-Guided Search for Deletion-Correcting Codes
Finding deletion-correcting codes of maximum size has been an open problem for over 70 years, even for a single deletion. In this paper, we propose a novel approach for constructing deletion-correcting codes. A code is a…
Large Language ModelLoopy Belief Propagation for Approximate Inference: An Empirical Study
Recently, researchers have demonstrated that loopy belief propagation - the use of Pearls polytree algorithm IN a Bayesian network WITH loops OF error- correcting codes.The most dramatic instance OF this IS the near Shan…
Interpreting Training Aspects of Deep-Learned Error-Correcting Codes
As new deep-learned error-correcting codes continue to be introduced, it is important to develop tools to interpret the designed codes and understand the training process. Prior work focusing on the deep-learned TurboAE …
Decoder