Mitigating Shortcuts in Language Models with Soft Label Encoding
Recent research has shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. In this work, we aim to answer the following research question: Can we reduce spurious correlations by modifying the ground truth labels of the training data? Specifically, we propose a simple yet effective debiasing framework, named Soft Label Encoding (SoftLE). We first train a teacher model with hard labels to determine each sample's degree of relying on shortcuts. We then add one dummy class to encode the shortcut degree, which is used to smooth other dimensions in the ground truth label to generate soft labels. This new ground truth label is used to train a more robust student model. Extensive experiments on two NLU benchmark tasks demonstrate that SoftLE significantly improves out-of-distribution generalization while maintaining satisfactory in-distribution accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingOut-of-Distribution GeneralizationSimilar Papers 제목 키워드 기반
Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning
Large language models (LLMs) have recently shown great potential for in-context learning, where LLMs learn a new task simply by conditioning on a few input-label pairs (prompts). Despite their potential, our understandin…
In-Context LearningMiMu: Mitigating Multiple Shortcut Learning Behavior of Transformers
Empirical Risk Minimization (ERM) models often rely on spurious correlations between features and labels during the learning process, leading to shortcut learning behavior that undermines robustness generalization perfor…
A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension
The issue of shortcut learning is widely known in NLP and has been an important research focus in recent years. Unintended correlations in the data enable models to easily solve tasks that were meant to exhibit advanced …
Machine Reading ComprehensionReading ComprehensionIdentifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models
Recently, NLP models have achieved remarkable progress across a variety of tasks; however, they have also been criticized for being not robust. Many robustness problems can be attributed to models exploiting spurious cor…
Differentiable Logic Programming to Mitigate Reasoning Shortcuts in Neurosymbolic Systems
Neurosymbolic (NeSy) systems integrate neural networks with logical reasoning to achieve both generalization and interpretability, but recent work has shown they are susceptible to shortcut reasoning behaviors. We propos…
Logical Reasoning