Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning
Even though BERT achieves successful performance improvements in various supervised learning tasks, applying BERT for unsupervised tasks still holds a limitation that it requires repetitive inference for computing contextual language representations. To resolve the limitation, we propose a novel deep bidirectional language model called Transformer-based Text Autoencoder (T-TA). The T-TA computes contextual language representations without repetition and has benefits of the deep bidirectional architecture like BERT. In run-time experiments on CPU environments, the proposed T-TA performs over six times faster than the BERT-based model in the reranking task and twelve times faster in the semantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks.
Code (1)
Tasks
CPULanguage ModelingLanguage ModellingRerankingSemantic SimilaritySemantic Textual SimilarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation
This work presents our ongoing research of unsupervised pretraining in neural machine translation (NMT). In our method, we initialize the weights of the encoder and decoder with two language models that are trained with …
DecoderLanguage ModelingLanguage ModellingMachine Translation+2Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation
This work presents our ongoing research of unsupervised pretraining in neural machine translation (NMT). In our method, we initialize the weights of the encoder and decoder with two language models that are trained with …
DecoderLanguage ModelingLanguage ModellingMachine Translation+2Contextual Salience for Fast and Accurate Sentence Vectors
Unsupervised vector representations of sentences or documents are a major building block for many language tasks such as sentiment classification. However, current methods are uninterpretable and slow or require large tr…
Document SummarizationGeneral ClassificationSentenceSentence Embeddings+2Transformer Based Language Models for Similar Text Retrieval and Ranking
Most approaches for similar text retrieval and ranking with long natural language queries rely at some level on queries and responses having words in common with each other. Recent applications of transformer-based neura…
Natural Language QueriesRetrievalSentenceText RetrievalA text autoencoder from transformer for fast encoding language representation
In recent years BERT shows apparent advantages and great potential in natural language processing tasks. However, both training and applying BERT requires intensive time and resources for computing contextual language re…
CPULanguage ModelingLanguage ModellingSemantic Similarity+2