DeLighT

2000년 도입 · 논문 1편에서 사용

DeLiGHT is a transformer architecture that delivers parameter efficiency improvements by (1) within each Transformer block using DExTra, a deep and light-weight transformation, allowing for the use of single-headed attention and bottleneck FFN layers and (2) across blocks using block-wise scaling, that allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output.

출처: DeLighT: Deep and Light-weight Transformer

소개 논문: DeLighT: Deep and Light-weight Transformer