Thick-Net: Parallel Network Structure for Sequential Modeling
Recurrent neural networks have been widely used in sequence learning tasks. In previous studies, the performance of the model has always been improved by either wider or deeper structures. However, the former becomes more prone to overfitting, while the latter is difficult to optimize. In this paper, we propose a simple new model named Thick-Net, by expanding the network from another dimension: thickness. Multiple parallel values are obtained via more sets of parameters in each hidden state, and the maximum value is selected as the final output among parallel intermediate outputs. Notably, Thick-Net can efficiently avoid overfitting, and is easier to optimize than the vanilla structures due to the large dropout affiliated with it. Our model is evaluated on four sequential tasks including adding problem, permuted sequential MNIST, text classification and language modeling. The results of these tasks demonstrate that our model can not only improve accuracy with faster convergence but also facilitate a better generalization ability.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage Modellingtext-classificationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Modeling Code-Switch Languages Using Bilingual Parallel Corpus
Language modeling is the technique to estimate the probability of a sequence of words. A bilingual language model is expected to model the sequential dependency for words across languages, which is difficult due to the i…
Bilingual Lexicon InductionLanguage ModelingLanguage ModellingTranslationLsrIF: Enhancing Logic-Structured Instruction Following of Large Language Models
Instruction following is critical for large language models, yet real-world instructions often involve multiple constraints with logical structures, such as parallel composition, sequential dependencies, and conditional …
Instruction FollowingParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models
Recurrent Neural Networks (RNNs) laid the foundation for sequence modeling, but their intrinsic sequential nature restricts parallel computation, creating a fundamental barrier to scaling. This has led to the dominance o…
Design of Resistive Frequency Selective Surface based Radar Absorbing Structure-A Deep Learning Approach
In this paper, deep learning-based approach for the design of radar absorbing structure using resistive frequency selective surface is proposed. In the present design, reflection coefficient is used as input of deep lear…
Deep LearningEnd-to-end differentiable design of geometric waveguide displays
Geometric waveguides are a promising architecture for optical see-through augmented reality displays, but their performance is severely bottlenecked by the difficulty of jointly optimizing non-sequential light transport …