Enhancing SLM via ChatGPT and Dataset Augmentation

2024-09-19 · Tom Pieper, Mohamad Ballout, Ulf Krumnack, Gunther Heidemann, Kai-Uwe Kühnberger

This paper explores the enhancement of small language models through strategic dataset augmentation via ChatGPT-3.5-Turbo, in the domain of Natural Language Inference (NLI). By employing knowledge distillation-based techniques and synthetic dataset augmentation, we aim to bridge the performance gap between large language models (LLMs) and small language models (SLMs) without the immense cost of human annotation. Our methods involve two forms of rationale generation--information extraction and informed reasoning--to enrich the ANLI dataset. We then fine-tune T5-Small on these augmented datasets, evaluating its performance against an established benchmark. Our findings reveal that the incorporation of synthetic rationales significantly improves the model's ability to comprehend natural language, leading to 1.3\% and 2.3\% higher classification accuracy, respectively, on the ANLI dataset, demonstrating the potential of leveraging LLMs for dataset augmentation. This approach not only enhances the performance of smaller models on complex tasks but also introduces a cost-effective method for fine-tuning smaller language models. By advancing our understanding of knowledge distillation and fine-tuning strategies, this work contributes to the ongoing effort to create more capable and efficient NLP systems.

📄 PDF Abstract BibTeX arXiv:2409.12599

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation Natural Language Inference

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Scaling Arabic Medical Chatbots Using Synthetic Data: Enhancing Generative AI with Synthetic Patient Records

2025-09-12 · Abdulrahman Allam, Seif Ahmed, Ali Hamdi, Khaled Shaban arxiv

The development of medical chatbots in Arabic is significantly constrained by the scarcity of large-scale, high-quality annotated datasets. While prior efforts compiled a dataset of 20,000 Arabic patient-doctor interacti…

Data Augmentation

3D-VirtFusion: Synthetic 3D Data Augmentation through Generative Diffusion Models and Controllable Editing

2024-08-25 · Shichao Dong, Ze Yang, Guosheng Lin

Data augmentation plays a crucial role in deep learning, enhancing the generalization and robustness of learning-based models. Standard approaches involve simple transformations like rotations and flips for generating ex…

Data Augmentation Diversity Few-Shot Learning Scene Understanding

ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection

2023-08-25 · Yihao Fang, Xianzhi Li, Stephen W. Thomas, Xiaodan Zhu

Open intent detection, a crucial aspect of natural language understanding, involves the identification of previously unseen intents in user-generated text. Despite the progress made in this field, challenges persist in h…

Data Augmentation Intent Detection Natural Language Understanding Open Intent Detection

Image, Text, and Speech Data Augmentation using Multimodal LLMs for Deep Learning: A Survey

2025-01-29 · Ranjan Sapkota, Shaina Raza, Maged Shoman, Achyut Paudel 외

In the past five years, research has shifted from traditional Machine Learning (ML) and Deep Learning (DL) approaches to leveraging Large Language Models (LLMs) , including multimodality, for data augmentation to enhance…

Data Augmentation Image Augmentation Text Augmentation

Medical Data Augmentation via ChatGPT: A Case Study on Medication Identification and Medication Event Classification

2023-06-10 · Shouvon Sarker, Lijun Qian, Xishuang Dong

The identification of key factors such as medications, diseases, and relationships within electronic health records and clinical notes has a wide range of applications in the clinical field. In the N2C2 2022 competitions…