Primer

2000년 도입 · 논문 14편에서 사용

Primer is a Transformer-based architecture that improves upon the Transformer architecture with two improvements found through neural architecture search: squared RELU activations in the feedforward block, and [depthwise convolutions]() added to the attention multi-head projections: resulting in a new module called Multi-DConv-Head-Attention.

출처: Primer: Searching for Efficient Transformers for Language Modeling

소개 논문: Primer: Searching for Efficient Transformers for Language Modeling