Does Knowledge Help General NLU? An Empirical Study
It is often observed in knowledge-centric tasks (e.g., common sense question and answering, relation classification) that the integration of external knowledge such as entity representation into language models can help provide useful information to boost the performance. However, it is still unclear whether this benefit can extend to general natural language understanding (NLU) tasks. In this work, we empirically investigated the contribution of external knowledge by measuring the end-to-end performance of language models with various knowledge integration methods. We find that the introduction of knowledge can significantly improve the results on certain tasks while having no adverse effects on other tasks. We then employ mutual information to reflect the difference brought by knowledge and a neural interpretation model to reveal how a language model utilizes external knowledge. Our study provides valuable insights and guidance for practitioners to equip NLP models with knowledge.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningLanguage ModelingLanguage ModellingNatural Language UnderstandingRelation ClassificationSimilar Papers 제목 키워드 기반
Knowledge Diffusion for Neural Dialogue Generation
End-to-end neural dialogue generation has shown promising results recently, but it does not employ knowledge to guide the generation and hence tends to generate short, general, and meaningless responses. In this paper, w…
Dialogue GenerationQuestion AnsweringTask-Oriented Dialogue SystemsKnowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering
A recent Nature Medicine study reports that general-purpose frontier LLMs outperform specialized retrieval-augmented clinical tools on medical benchmarks, and that retrieval can hurt strong models. We ask the natural fol…
Question AnsweringAn Empirical Study on Finding Spans
We present an empirical study on methods for span finding, the selection of consecutive tokens in text for some downstream tasks. We focus on approaches that can be employed in training end-to-end information extraction …
PredictionPatching as Translation: the Data and the Metaphor
Machine Learning models from other fields, like Computational Linguistics, have been transplanted to Software Engineering tasks, often quite successfully. Yet a transplanted model's initial success at a given task does n…
General KnowledgeProgram RepairTranslationEmpirical Study of Mix-based Data Augmentation Methods in Physiological Time Series Data
Data augmentation is a common practice to help generalization in the procedure of deep model training. In the context of physiological time series classification, previous research has primarily focused on label-invarian…
Data AugmentationTime SeriesTime Series Classification