Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain
Engineering knowledge-based (or expert) systems require extensive manual effort and domain knowledge. As Large Language Models (LLMs) are trained using an enormous amount of cross-domain knowledge, it becomes possible to automate such engineering processes. This paper presents an empirical automation and semi-automation framework for domain knowledge distillation using prompt engineering and the LLM ChatGPT. We assess the framework empirically in the autonomous driving domain and present our key observations. In our implementation, we construct the domain knowledge ontology by "chatting" with ChatGPT. The key finding is that while fully automated domain ontology construction is possible, human supervision and early intervention typically improve efficiency and output quality as they lessen the effects of response randomness and the butterfly effect. We, therefore, also develop a web-based distillation assistant enabling supervision and flexible intervention at runtime. We hope our findings and tools could inspire future research toward revolutionizing the engineering of knowledge-based systems across application domains.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingKnowledge DistillationLanguage ModelingLanguage ModellingLarge Language ModelPrompt EngineeringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation
We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both domain adaptation and knowledge distillati…
Domain AdaptationKnowledge DistillationMachine TranslationTranslationDistilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual
Vision-language models (VLMs) exhibit uneven performance across languages, a problem that is often exacerbated when the model size is reduced. While Knowledge distillation (KD) demonstrates promising results in transferr…
Knowledge DistillationModel CompressionScaling Laws for Task-Specific LLM Distillation
Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives e…
General KnowledgeHow to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge in a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing…
Adversarial RobustnessAllData AugmentationDomain Generalization+3How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one. Even though KD has shown promise on a wide range of Natural Language Processing…
Adversarial RobustnessAllData AugmentationDomain Generalization+3