APAM: Adaptive Pre-training and Adaptive Meta Learning in Language Model for Noisy Labels and Long-tailed Learning
Practical natural language processing (NLP) tasks are commonly long-tailed with noisy labels. Those problems challenge the generalization and robustness of complex models such as Deep Neural Networks (DNNs). Some commonly used resampling techniques, such as oversampling or undersampling, could easily lead to overfitting. It is growing popular to learn the data weights leveraging a small amount of metadata. Besides, recent studies have shown the advantages of self-supervised pre-training, particularly to the under-represented data. In this work, we propose a general framework to handle the problem of both long-tail and noisy labels. The model is adapted to the domain of problems in a contrastive learning manner. The re-weighting module is a feed-forward network that learns explicit weighting functions and adapts weights according to metadata. The framework further adapts weights of terms in the loss function through a combination of the polynomial expansion of cross-entropy loss and focal loss. Our extensive experiments show that the proposed framework consistently outperforms baseline methods. Lastly, our sensitive analysis emphasizes the capability of the proposed framework to handle the long-tailed problem and mitigate the negative impact of noisy labels.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningLanguage ModelingLanguage ModellingMeta-LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Making Cross-Domain Recommendations by Associating Disjoint Users and Items Through the Affective Aware Pseudo Association Method
This paper utilizes an ingenious text-based affective aware pseudo association method (AAPAM) to link disjoint users and items across different information domains and leverage them to make cross-domain content-based and…
Collaborative FilteringInformation RetrievalRetrievalMetaAdapt: Domain Adaptive Few-Shot Misinformation Detection via Meta Learning
With emerging topics (e.g., COVID-19) on social media as a source for the spreading misinformation, overcoming the distributional shifts between the original training domain (i.e., source domain) and such target domains …
Meta-LearningMisinformationTransfer LearningAdaptive Constraint Propagation: Scaling Structured Inference for Large Language Models via Meta-Reinforcement Learning
Large language models increasingly require structured inference, from JSON schema enforcement to multi-lingual parsing, where outputs must satisfy complex constraints. We introduce MetaJuLS, a meta-reinforcement learning…
Reinforcement LearningDomain AdaptationMeta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks
Federated learning (FL) is susceptible to a range of security threats. Although various defense mechanisms have been proposed, they are typically non-adaptive and tailored to specific types of attacks, leaving them insuf…
Federated LearningMeta-LearningModel PoisoningReinforcement Learning (RL)Bitwidth-Adaptive Quantization-Aware Neural Network Training: A Meta-Learning Approach
Deep neural network quantization with adaptive bitwidths has gained increasing attention due to the ease of model deployment on various platforms with different resource budgets. In this paper, we propose a meta-learning…
Few-Shot LearningMeta-LearningQuantization