Memory Augmented Language Models through Mixture of Word Experts

2023-11-15 · Cicero Nogueira dos santos, James Lee-Thorp, Isaac Noble, Chung-Ching Chang, David Uthus

Scaling up the number of parameters of language models has proven to be an effective approach to improve performance. For dense models, increasing model size proportionally increases the model's computation footprint. In this work, we seek to aggressively decouple learning capacity and FLOPs through Mixture-of-Experts (MoE) style models with large knowledge-rich vocabulary based routing functions and experts. Our proposed approach, dubbed Mixture of Word Experts (MoWE), can be seen as a memory augmented model, where a large set of word-specific experts play the role of a sparse memory. We demonstrate that MoWE performs significantly better than the T5 family of models with similar number of FLOPs in a variety of NLP tasks. Additionally, MoWE outperforms regular MoE models on knowledge intensive tasks and has similar performance to more complex memory augmented approaches that often require to invoke custom mechanisms to search the sparse memory.

📄 PDF Abstract BibTeX arXiv:2311.10768

Code (0)

등록된 구현이 없습니다.

Tasks

Mixture-of-Experts

Methods 이 논문이 사용한 방법론

Gated Linear Unit A Gated Linear Unit, or GLU computes: $$ \mathrm{GLU}(a, b) = a \otimes \sigma(b) $$ It is used in natural language processing architectures, for example the Gated CNN,…

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Residual Connection 설명 없음

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts

2026-01-05 · Boxuan Lyu, Soichiro Murakami, Hidetaka Kamigaito, Peinan Zhang arxiv

Mixture-of-Experts (MoE) architectures scale large language models efficiently by employing a parametric ``router'' to dispatch tokens to a sparse subset of experts. Typically, this router is trained once and then frozen…

Neural Machine Translation with Key-Value Memory-Augmented Attention

2018-06-29 · Fandong Meng, Zhaopeng Tu, Yong Cheng, Haiyang Wu 외

Although attention-based Neural Machine Translation (NMT) has achieved remarkable progress in recent years, it still suffers from issues of repeating and dropping translations. To alleviate these issues, we propose a nov…

Decoder Machine Translation NMT Sentence+1

Frustratingly Short Attention Spans in Neural Language Modeling

2017-02-15 · Michał Daniluk, Tim Rocktäschel, Johannes Welbl, Sebastian Riedel

Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiabl…

Language Modeling Language Modelling

ReM-MoA: Reasoning Memory Sustains Mixture-of-Agents Scaling

2026-06-23 · Heng Ping, Arijit Bhattacharjee, Peiyu Zhang, Shixuan Li 외 arxiv

Mixture-of-Agents (MoA) architectures improve inference-time scaling by organizing multiple LLM agents into layered reasoning pipelines. However, existing MoA variants fail to sustain gains as depth increases, exhibiting…

Formal Logic

Gated Word-Character Recurrent Language Model

2016-06-06 · EMNLP 2016 11 · Yasumasa Miyamoto, Kyunghyun Cho

We introduce a recurrent neural network language model (RNN-LM) with long short-term memory (LSTM) units that utilizes both character-level and word-level inputs. Our model has a gate that adaptively finds the optimal mi…

Language Modeling Language Modelling model