Interpretable Sentence Representation with Variational Autoencoders and Attention
In this thesis, we develop methods to enhance the interpretability of recent representation learning techniques in natural language processing (NLP) while accounting for the unavailability of annotated data. We choose to leverage Variational Autoencoders (VAEs) due to their efficiency in relating observations to latent generative factors and their effectiveness in data-efficient learning and interpretable representation learning. As a first contribution, we identify and remove unnecessary components in the functioning scheme of semi-supervised VAEs making them faster, smaller and easier to design. Our second and main contribution is to use VAEs and Transformers to build two models with inductive bias to separate information in latent representations into understandable concepts without annotated data. The first model, Attention-Driven VAE (ADVAE), is able to separately represent and control information about syntactic roles in sentences. The second model, QKVAE, uses separate latent variables to form keys and values for its Transformer decoder and is able to separate syntactic and semantic information in its neural representations. In transfer experiments, QKVAE has competitive performance compared to supervised models and equivalent performance to a supervised model using 50K annotated samples. Additionally, QKVAE displays improved syntactic role disentanglement capabilities compared to ADVAE. Overall, we demonstrate that it is possible to enhance the interpretability of state-of-the-art deep learning architectures for language modeling with unannotated data in situations where text data is abundant but annotations are scarce.
Code (0)
등록된 구현이 없습니다.
Tasks
DisentanglementInductive BiasLanguage ModelingLanguage ModellingRepresentation LearningSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation
The encoder-decoder dialog model is one of the most prominent methods used to build dialog systems in complex domains. Yet it is limited because it cannot output interpretable actions as in traditional systems, which hin…
DecoderDialogue GenerationDialogue InterpretationRepresentation Learning+3Isotropic Contextual Representations through Variational Regularization
Contextual language representations achieve state-of-the-art performance across various natural language processing tasks. However, these representations have been shown to suffer from the degeneration problem, i.e. they…
DecoderSentenceInducing Interpretable Representations with Variational Autoencoders
We develop a framework for incorporating structured graphical models in the \emph{encoders} of variational autoencoders (VAEs) that allows us to induce interpretable representations through approximate variational infere…
General ClassificationVariational InferenceHierarchically-Structured Variational Autoencoders for Long Text Generation
Variational autoencoders (VAEs) have received much attention recently as an end-to-end architecture for text generation. Existing methods primarily focus on synthesizing relatively short sentences (with less than twenty …
DecoderSentenceText GenerationText Modeling with Syntax-Aware Variational Autoencoders
Syntactic information contains structures and rules about how text sentences are arranged. Incorporating syntax into text modeling methods can potentially benefit both representation learning and generation. Variational …
Representation Learning