Mosaic-IT: Free Compositional Data Augmentation Improves Instruction Tuning
Finetuning large language models with a variety of instruction-response pairs has enhanced their capability to understand and follow instructions. Current instruction tuning primarily relies on teacher models or human intervention to generate and refine the instructions and responses for training, which are costly, non-sustainable, and may lack diversity. In this paper, we introduce Mosaic Instruction Tuning (Mosaic-IT), a human/model-free compositional data augmentation method that can efficiently create rich and diverse augmentations from existing instruction tuning data to enhance the LLMs. Mosaic-IT randomly concatenates multiple instruction data into one and trains the model to produce the corresponding responses with predefined higher-level meta-instructions to strengthen its multi-step instruction-following and format-following skills. Our extensive evaluations demonstrate a superior performance and training efficiency of Mosaic-IT, which achieves consistent performance improvements over various benchmarks and a $80\%$ reduction in training costs compared with original instruction tuning. Our codes and data are available at https://github.com/tianyi-lab/Mosaic-IT.
Code (3)
Tasks
Data AugmentationDiversityInstruction FollowingSimilar Papers 제목 키워드 기반
Memory Mosaics
Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest. Like transformers, memory mosaics possess compositional capabilities and in-context learning capabilities. …
DisentanglementIn-Context LearningLanguage ModelingLanguage ModellingSelect-Mosaic: Data Augmentation Method for Dense Small Object Scenes
Data augmentation refers to the process of applying a series of transformations or expansions to original data to generate new samples, thereby increasing the diversity and quantity of the data, effectively improving the…
Data AugmentationDiversityobject-detectionObject Detection+1Mosaic Representation Learning for Self-supervised Visual Pre-training
Self-supervised learning has achieved significant success in learning visual representations without the need for manual annotation. To obtain generalizable representations, a meticulously designed data augmentation stra…
Data AugmentationRepresentation LearningSelf-Supervised LearningMosaic: Compositional Multi-Concept Erasure via Vector Field Blending
Concept erasure has emerged as a key research direction for ensuring safe and ethical image synthesis in Text-to-Image (T2I) models. While existing studies have explored concept erasure across multiple concepts, they typ…
AudioMosaic: Contrastive Masked Audio Representation Learning
Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive…
Self-Supervised LearningRepresentation LearningContrastive Learning