PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
In this paper, we introduce PanGu-Bot, a Chinese pre-trained open-domain dialogue generation model based on a large pre-trained language model (PLM) PANGU-alpha (Zeng et al.,2021). Different from other pre-trained dialogue models trained over a massive amount of dialogue data from scratch, we aim to build a powerful dialogue model with relatively fewer data and computation costs by inheriting valuable language capabilities and knowledge from PLMs. To this end, we train PanGu-Bot from the large PLM PANGU-alpha, which has been proven well-performed on a variety of Chinese natural language tasks. We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety. We show that PanGu-Bot outperforms state-of-the-art Chinese dialogue systems (CDIALGPT (Wang et al., 2020), EVA (Zhou et al., 2021), EVA2.0 (Gu et al., 2022)) w.r.t. the above three aspects. We also demonstrate that PanGu-Bot can be easily deployed to generate emotional responses without further training. Throughout our empirical analysis, we also point out that the PanGu-Bot response quality, knowledge correctness, and safety are still far from perfect, and further explorations are indispensable to building reliable and smart dialogue systems. Our model and code will be available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/PanGu-Bot soon.
Code (2)
Tasks
Dialogue GenerationLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural l…
Cloze (multi-choices) (Few-Shot)Cloze (multi-choices) (One-Shot)Cloze (multi-choices) (Zero-Shot)Common Sense Reasoning (Few-Shot)+17PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Asce…
Code GenerationLanguage ModelingLanguage ModellingMachine Translation+3PanGu-Coder: Program Synthesis with Function-Level Language Modeling
We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i.e. the synthesis of programming language solutions given a natural language problem de…
Code GenerationDecoderLanguage ModelingLanguage Modelling+2PanGu-$π$: Enhancing Language Model Architectures via Nonlinearity Compensation
The recent trend of large language models (LLMs) is to increase the scale of both model size (\aka the number of parameters) and dataset to achieve better generative ability, which is definitely proved by a lot of work s…
Language ModelingLanguage ModellingDon't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments
A key missing capacity of current language models (LMs) is grounding to real-world environments. Most existing work for grounded language understanding uses LMs to directly generate plans that can be executed in the envi…
In-Context LearningKnowledge Base Question AnsweringQuestion Answeringvalid