Lightweight Text Classifier using Sinusoidal Positional Encoding
Large and complex models have recently been developed that require many parameters and much time to solve various problems in natural language processing. This paper explores an efficient way to avoid models being too complicated and ensure nearly equal performance to models showing the state-of-the-art. We propose a single convolutional neural network (CNN) using the sinusoidal positional encoding (SPE) in text classification. The SPE provides useful position information of a word and can construct a more efficient model architecture than before in a CNN-based approach. Our model can significantly reduce the parameter size (at least 67{\%}) and training time (up to 85{\%}) while maintaining similar performance to the CNN-based approach on multiple benchmark datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Positiontext-classificationText ClassificationSimilar Papers 제목 키워드 기반
CoPE: A Lightweight Complex Positional Encoding
Recent studies have demonstrated the effectiveness of position encoding in transformer architectures. By incorporating positional information, this approach provides essential guidance for modeling dependencies between e…
The Impact of Positional Encodings on Multilingual Compression
In order to preserve word-order information in a non-autoregressive setting, transformer architectures tend to include positional knowledge, by (for instance) adding positional encodings to token embeddings. Several modi…
Inductive BiasLoPE: Learnable Sinusoidal Positional Encoding for Improving Document Transformer Model
Positional encoding plays a key role in Transformer-based architecture, which is to indicate and embed token sequential order information. Understanding documents with unreliable reading order information is a real chall…
document understandingAlternative positional encoding functions for neural transformers
A key module in neural transformer-based deep architectures is positional encoding. This module enables a suitable way to encode positional information as input for transformer neural layers. This success has been rooted…
Spiking Sequence Machines and Transformers
Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distrib…