Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictability on online human language comprehension. However, two recently developed recurrent model architectures, RWKV and Mamba, appear to perform natural language tasks comparably to or better than transformers of equivalent scale. In this paper, we show that contemporary recurrent models are now also able to match - and in some cases, exceed - the performance of comparably sized transformers at modeling online human language comprehension. This suggests that transformer language models are not uniquely suited to this task, and opens up new directions for debates about the extent to which architectural features of language models make them better or worse models of human language comprehension.
Code (1)
Tasks
MambaSimilar Papers 제목 키워드 기반
Leveraging Transformers for StarCraft Macromanagement Prediction
Inspired by the recent success of transformers in natural language processing and computer vision applications, we introduce a transformer-based neural architecture for two key StarCraft II (SC2) macromanagement tasks: g…
PredictionStarcraftStarcraft IITransfer LearningShould You Fine-Tune BERT for Automated Essay Scoring?
Most natural language processing research now recommends large Transformer-based models with fine-tuning for supervised classification tasks; older strategies like bag-of-words features and linear models have fallen out …
Automated Essay ScoringExpert-augmented actor-critic for ViZDoom and Montezumas Revenge
We propose an expert-augmented actor-critic algorithm, which we evaluate on two environments with sparse rewards: Montezumas Revenge and a demanding maze from the ViZDoom suite. In the case of Montezumas Revenge, an agen…
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard T…
Fallen Angel Bonds Investment and Bankruptcy Predictions Using Manual Models and Automated Machine Learning
The primary aim of this research was to find a model that best predicts which fallen angel bonds would either potentially rise up back to investment grade bonds and which ones would fall into bankruptcy. To implement the…
feature selection