A text autoencoder from transformer for fast encoding language representation
In recent years BERT shows apparent advantages and great potential in natural language processing tasks. However, both training and applying BERT requires intensive time and resources for computing contextual language representations, which hinders its universality and applicability. To overcome this bottleneck, we propose a deep bidirectional language model by using window masking mechanism at attention layer. This work computes contextual language representations without random masking as does in BERT and maintains the deep bidirectional architecture like BERT. To compute the same sentence representation, our method shows O(n) complexity less compared to other transformer-based models with O($n^2$). To further demonstrate its superiority, computing context language representations on CPU environments is conducted, by using the embeddings from the proposed method, logistic regression shows much higher accuracy in terms of SMS classification. Moverover, the proposed method also achieves significant higher performance in semantic similarity tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
CPULanguage ModelingLanguage ModellingSemantic SimilaritySemantic Textual SimilaritySentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Encoding Musical Style with Transformer Autoencoders
We consider the problem of learning high-level controls over the global structure of generated sequences, particularly in the context of symbolic music generation with complex language models. In this work, we present th…
Music GenerationLearning the Koopman Operator using Attention Free Transformers
Learning Koopman operators with autoencoders enables linear prediction in a latent space, but long-horizon rollouts often drift off the learned manifold, leading to phase and amplitude errors on systems with switching, c…
Recurrent autoencoder with sequence-aware encoding
Recurrent Neural Networks (RNN) received a vast amount of attention last decade. Recently, the architectures of Recurrent AutoEncoders (RAE) found many applications in practice. RAE can extract the semantically valuable …
Fast and Accurate Transformer-based Translation with Character-Level Encoding and Subword-Level Decoding
The Transformer translation model is fast to train and achieves state-of-the-art results for various translation tasks. However, unknown input words at test time remain a challenge for the Transformer, especially when …
DecoderTranslationSpatio-Temporal Encoding of Brain Dynamics with Surface Masked Autoencoders
The development of robust and generalisable models for encoding the spatio-temporal dynamics of human brain activity is crucial for advancing neuroscientific discoveries. However, significant individual variation in the …
Transfer Learning