Meta-Learning Fast Weight Language Models
Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens and substantially improves LM performance. However, it requires over 3x more compute than standard inference. We present Fast Weight Layers (FWLs), a neural component that provides the benefits of dynamic evaluation much more efficiently by expressing gradient updates as linear attention. A key improvement over dynamic evaluation is that FWLs can also be applied at training time so the model learns to make good use of gradient updates. FWLs can easily be added on top of existing transformer models, require relatively little extra compute or memory to run, and significantly improve language modeling perplexity.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMeta-LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
Training a deep neural network requires a large amount of single-task data and involves a long time-consuming optimization phase. This is not scalable to complex, realistic environments with new unexpected changes. Human…
Incremental LearningInductive BiasLanguage ModellingMeta-Learning+1Meta Learning Backpropagation And Improving It
Many concepts have been proposed for meta learning with neural networks (NNs), e.g., NNs that learn to reprogram fast weights, Hebbian plasticity, learned learning rules, and meta recurrent NNs. Our Variable Shared Meta …
Meta-LearningLearning Associative Inference Using Fast Weight Memory
Humans can quickly associate stimuli to solve problems in novel contexts. Our novel neural network model learns state representations of facts that can be composed to perform such associative inference. To this end, we a…
Language ModellingMeta Reinforcement LearningQuestion Answeringreinforcement-learning+1Metalearning with Hebbian Fast Weights
We unify recent neural approaches to one-shot learning with older ideas of associative memory in a model for metalearning. Our model learns jointly to represent data and to bind class labels to representations in a singl…
One-Shot LearningMetaScope - Fast and accurate identification of microbes in metagenomic sequencing data
MetaScope is a fast and accurate tool for analyzing (host-associated) metagenome datasets. Sequence alignment of reads against the host genome (if requested) and against microbial Genbank is performed using a new DNA ali…