Exploring the Robustness of In-Context Learning with Noisy Labels
Recently, the mysterious In-Context Learning (ICL) ability exhibited by Transformer architectures, especially in large language models (LLMs), has sparked significant research interest. However, the resilience of Transformers' in-context learning capabilities in the presence of noisy samples, prevalent in both training corpora and prompt demonstrations, remains underexplored. In this paper, inspired by prior research that studies ICL ability using simple function classes, we take a closer look at this problem by investigating the robustness of Transformers against noisy labels. Specifically, we first conduct a thorough evaluation and analysis of the robustness of Transformers against noisy labels during in-context learning and show that they exhibit notable resilience against diverse types of noise in demonstration labels. Furthermore, we delve deeper into this problem by exploring whether introducing noise into the training set, akin to a form of data augmentation, enhances such robustness during inference, and find that such noise can indeed improve the robustness of ICL. Overall, our fruitful analysis and findings provide a comprehensive understanding of the resilience of Transformer models against label noises during ICL and provide valuable insights into the research on Transformers in natural language processing. Our code is available at https://github.com/InezYu0928/in-context-learning.
Code (1)
Tasks
Data AugmentationIn-Context LearningLearning with noisy labelsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring Loss Design Techniques For Decision Tree Robustness To Label Noise
In the real world, data is often noisy, affecting not only the quality of features but also the accuracy of labels. Current research on mitigating label errors stems primarily from advances in deep learning, and a gap ex…
Deep LearningExploring the Role of Diversity in Example Selection for In-Context Learning
In-Context Learning (ICL) has gained prominence due to its ability to perform tasks without requiring extensive training data and its robustness to noisy labels. A typical ICL workflow involves selecting localized exampl…
DiversityIn-Context LearningRerankingGraph Neural Networks with Coarse- and Fine-Grained Division for Mitigating Label Sparsity and Noise
Graph Neural Networks (GNNs) have gained considerable prominence in semi-supervised learning tasks in processing graph-structured data, primarily owing to their message-passing mechanism, which largely relies on the avai…
Graph ReconstructionNode ClassificationAnnot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension
Training with noisy class labels impairs neural networks' generalization performance. In this context, mixup is a popular regularization technique to improve training robustness by making memorizing false class labels mo…
How does self-supervised pretraining improve robustness against noisy labels across various medical image classification datasets?
Noisy labels can significantly impact medical image classification, particularly in deep learning, by corrupting learned features. Self-supervised pretraining, which doesn't rely on labeled data, can enhance robustness a…
Contrastive Learningimage-classificationImage ClassificationMedical Image Classification+1