LLaMA

2000년 도입 · 논문 1,062편에서 사용

LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were subsequently proposed. The main difference with the original architecture are listed below. - RMSNorm normalizing function is used to improve the training stability, by normalizing the input of each transformer sub-layer, instead of normalizing the output. - The ReLU non-linearity is replaced by the SwiGLU activation function to improve performance. - Absolute positional embeddings are removed and instead rotary positional embeddings (RoPE) are added at each layer of the network.

출처: LLaMA: Open and Efficient Foundation Language Models

소개 논문: LLaMA: Open and Efficient Foundation Language Models