Discrete Autoencoders for Sequence Models
Recurrent models for sequences have been recently successful at many tasks, especially for language modeling and machine translation. Nevertheless, it remains challenging to extract good representations from these models. For instance, even though language has a clear hierarchical structure going from characters through words to sentences, it is not apparent in current language models. We propose to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space. In order to propagate gradients though this discrete representation we introduce an improved semantic hashing technique. We show that this technique performs well on a newly proposed quantitative efficiency measure. We also analyze latent codes produced by the model showing how they correspond to words and phrases. Finally, we present an application of the autoencoder-augmented model to generating diverse translations.
Code (2)
Tasks
Language ModelingLanguage ModellingMachine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model
The Sign Language Production (SLP) project aims to automatically translate spoken languages into sign sequences. Our approach focuses on the transformation of sign gloss sequences into their corresponding sign pose seque…
DenoisingQuantizationSign Language ProductionAdversarially Regularized Autoencoders
Deep latent variable models, trained using variational autoencoders or generative adversarial networks, are now a key technique for representation learning of continuous structures. However, applying similar methods to d…
Representation LearningStyle TransferTowards a better understanding of Vector Quantized Autoencoders
Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent vari…
Knowledge DistillationMachine TranslationTranslationEXCODER: EXplainable Classification Of DiscretE time series Representations
Deep learning has significantly improved time series classification, yet the lack of explainability in these models remains a major challenge. While Explainable AI (XAI) techniques aim to make model decisions more transp…
Time Series ClassificationTime Series AnalysisArch-LLM: Taming LLMs for Neural Architecture Generation via Unsupervised Discrete Representation Learning
Unsupervised representation learning has been widely explored across various modalities, including neural architectures, where it plays a key role in downstream applications like Neural Architecture Search (NAS). These m…
Large Language ModelNeural Architecture SearchRepresentation Learning