GPT-NeoX

2000년 도입 · 논문 11편에서 사용

GPT-NeoX is an autoregressive transformer decoder model whose architecture largely follows that of GPT-3, with a few notable deviations. The model has 20 billion parameters with 44 layers, a hidden dimension size of 6144, and 64 heads. The main difference with GPT-3 is the change in tokenizer, the addition of Rotary Positional Embeddings, the parallel computation of attention and feed-forward layers, and a different initialization scheme and hyperparameters.

출처: GPT-NeoX-20B: An Open-Source Autoregressive Language Model

소개 논문: GPT-NeoX-20B: An Open-Source Autoregressive Language Model