Symbolic Autoencoding for Self-Supervised Sequence Learning
Traditional language models, adept at next-token prediction in text sequences, often struggle with transduction tasks between distinct symbolic systems, particularly when parallel data is scarce. Addressing this issue, we introduce \textit{symbolic autoencoding} ($\Sigma$AE), a self-supervised framework that harnesses the power of abundant unparallel data alongside limited parallel data. $\Sigma$AE connects two generative models via a discrete bottleneck layer and is optimized end-to-end by minimizing reconstruction loss (simultaneously with supervised loss for the parallel data), such that the sequence generated by the discrete bottleneck can be read out as the transduced input sequence. We also develop gradient-based methods allowing for efficient self-supervised sequence learning despite the discreteness of the bottleneck. Our results demonstrate that $\Sigma$AE significantly enhances performance on transduction tasks, even with minimal parallel data, offering a promising solution for weakly supervised learning scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
Weakly-supervised LearningSimilar Papers 제목 키워드 기반
Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring
Recent work on unsupervised speech segmentation has used self-supervised models with phone and word segmentation modules that are trained jointly. This paper instead revisits an older approach to word segmentation: botto…
Acoustic Unit DiscoverySegmentationUnsupervised Learning of Neurosymbolic Encoders
We present a framework for the unsupervised learning of neurosymbolic encoders, which are encoders obtained by composing neural networks with symbolic programs from a domain-specific language. Our framework naturally inc…
DecoderProgram SynthesisSports AnalyticsRepresentation Learning for Sequence Data with Deep Autoencoding Predictive Components
We propose Deep Autoencoding Predictive Components (DAPC) -- a self-supervised representation learning method for sequence data, based on the intuition that useful representations of sequence data should exhibit a simple…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Contrastive LearningRepresentation Learning+2Speech segmentation with a neural encoder model of working memory
We present the first unsupervised LSTM speech segmenter as a cognitive model of the acquisition of words from unsegmented input. Cognitive biases toward phonological and syntactic predictability in speech are rooted in t…
DecoderRobust Human Trajectory Prediction via Self-Supervised Skeleton Representation Learning
Human trajectory prediction plays a crucial role in applications such as autonomous navigation and video surveillance. While recent works have explored the integration of human skeleton sequences to complement trajectory…
Representation LearningTrajectory Prediction