GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads) number of key-value heads. We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.
Code (4)
Tasks
DecoderLanguage ModelingLanguage ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
UP-DETR: Unsupervised Pre-training for Object Detection with Transformers
DEtection TRansformer (DETR) for object detection reaches competitive performance compared with Faster R-CNN via a transformer encoder-decoder architecture. However, trained with scratch transformers, DETR needs large-sc…
DecoderMulti-Task Learningobject-detectionObject Detection+2Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data
Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across …
Transfer LearningTransformer with Fourier Integral Attentions
Multi-head attention empowers the recent success of transformers, the state-of-the-art models that have achieved remarkable success in sequence modeling and beyond. These attention mechanisms compute the pairwise dot pro…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+1Textual Query-Driven Mask Transformer for Domain Generalized Segmentation
In this paper, we introduce a method to tackle Domain Generalized Semantic Segmentation (DGSS) by utilizing domain-invariant semantic knowledge from text embeddings of vision-language models. We employ the text embedding…
Domain GeneralizationObjectSemantic SegmentationGroupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection
Visual Relationship Detection (VRD) has seen significant advancements with Transformer-based architectures recently. However, we identify two key limitations in a conventional label assignment for training Transformer-ba…
RelationRelationship DetectionScene Graph GenerationVisual Relationship Detection