PAR Transformer

2000년 도입 · 논문 1편에서 사용

PAR Transformer is a Transformer model that uses 63% fewer self-attention blocks, replacing them with feed-forward blocks, while retaining test accuracies. It is based on the Transformer-XL architecture and uses neural architecture search to find an an efficient pattern of blocks in the transformer architecture.

출처: Pay Attention when Required

소개 논문: Pay Attention when Required