Weighted Training for Cross-Task Learning
In this paper, we introduce Target-Aware Weighted Training (TAWT), a weighted training algorithm for cross-task learning based on minimizing a representation-based task distance between the source and target tasks. We show that TAWT is easy to implement, is computationally efficient, requires little hyperparameter tuning, and enjoys non-asymptotic learning-theoretic guarantees. The effectiveness of TAWT is corroborated through extensive experiments with BERT on four sequence tagging tasks in natural language processing (NLP), including part-of-speech (PoS) tagging, chunking, predicate detection, and named entity recognition (NER). As a byproduct, the proposed representation-based task distance allows one to reason in a theoretically principled way about several critical aspects of cross-task learning, such as the choice of the source data and the impact of fine-tuning.
Code (1)
Tasks
Chunkingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERPart-Of-Speech TaggingPOSPOS TaggingPredicate DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FW-Shapley: Real-time Estimation of Weighted Shapley Values
Fair credit assignment is essential in various machine learning (ML) applications, and Shapley values have emerged as a valuable tool for this purpose. However, in critical ML applications such as data valuation and feat…
Data ValuationAligning Validation with Deployment in Spatial Prediction: Target-Weighted Cross-Validation
Reliable estimation of predictive performance is essential for spatial environmental modeling, where machine-learning models are used to generate maps from unevenly distributed observations. Standard cross-validation (CV…
Comparing Deep Learning strategies for paired but unregistered multimodal segmentation of the liver in T1 and T2-weighted MRI
We address the problem of multimodal liver segmentation in paired but unregistered T1 and T2-weighted MR images. We compare several strategies described in the literature, with or without multi-task training, with or wit…
Liver SegmentationParameter Averaging in Link Prediction
Ensemble methods are widely employed to improve generalization in machine learning. This has also prompted the adoption of ensemble learning for the knowledge graph embedding (KGE) models in performing link prediction. T…
Knowledge Graph EmbeddingEnsemble LearningLink PredictionQ3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
Parameter-efficient training based on low-rank optimization has become a highly successful tool for fine-tuning large deep learning models. However, these methods often fail for low-rank pre-training, where simultaneousl…