Curriculum Learning for Natural Language Understanding
With the great success of pre-trained language models, the pretrain-finetune paradigm now becomes the undoubtedly dominant solution for natural language understanding (NLU) tasks. At the fine-tune stage, target task data is usually introduced in a completely random order and treated equally. However, examples in NLU tasks can vary greatly in difficulty, and similar to human learning procedure, language models can benefit from an easy-to-difficult curriculum. Based on this idea, we propose our Curriculum Learning approach. By reviewing the trainset in a crossed way, we are able to distinguish easy examples from difficult ones, and arrange a curriculum for language models. Without any manual model architecture design or use of external data, our Curriculum Learning approach obtains significant and universal performance improvements on a wide range of NLU tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingSimilar Papers 제목 키워드 기반
Curriculum learning for language modeling
Language Models like ELMo and BERT have provided robust representations of natural language, which serve as the language understanding component for a diverse range of downstream tasks.Curriculum learning is a method tha…
Language ModelingLanguage ModellingMachine TranslationTranslationYour Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding
Curriculum learning is a widely adopted training strategy in natural language processing (NLP), where models are exposed to examples organized by increasing difficulty to enhance learning efficiency and performance. Howe…
Natural Language UnderstandingMulti-class ClassificationUser Adaptive Language Learning Chatbots with a Curriculum
Along with the development of systems for natural language understanding and generation, dialog systems have been widely adopted for language learning and practicing. Many current educational dialog systems perform chitc…
Natural Language UnderstandingCurriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
In the age of large transformer language models, linguistic evaluation play an important role in diagnosing models' abilities and limitations on natural language understanding. However, current evaluation methods show so…
Language ModelingLanguage ModellingNatural Language UnderstandingCurriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
In the age of large transformer language models, linguistic benchmarks play an important role in diagnosing models' abilities and limitations on natural language understanding. However, current benchmarks show some signi…
Language ModelingLanguage ModellingNatural Language Understanding