Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning
Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. In this work, we experiment with 130 source-target task combinations and demonstrate that the transfer performance exhibits severe variance across different source tasks and training seeds, highlighting the crucial role of intermediate-task selection in a broader context. We compare four representative task selection methods in a unified setup, focusing on their effectiveness and consistency. Compared to embedding-free methods and text embeddings, task embeddings constructed from fine-tuned weights can better estimate task transferability by improving task prediction scores from 2.59% to 3.96%. Despite their strong performance, we observe that the task embeddings do not consistently demonstrate superiority for tasks requiring reasoning abilities. Furthermore, we introduce a novel method that measures pairwise token similarity using maximum inner product search, leading to the highest performance in task prediction. Our findings suggest that token-wise similarity is better predictive for predicting transferability compared to averaging weights.
Code (1)
Tasks
Transfer LearningSimilar Papers 제목 키워드 기반
Exploring Cycle Consistency Learning in Interactive Volume Segmentation
Automatic medical volume segmentation often lacks clinical accuracy, necessitating further refinement. In this work, we interactively approach medical volume segmentation via two decoupled modules: interaction-to-segment…
SegmentationExploring Neural Models for Parsing Natural Language into First-Order Logic
Semantic parsing is the task of obtaining machine-interpretable representations from natural language text. We consider one such formal representation - First-Order Logic (FOL) and explore the capability of neural models…
DecoderSemantic ParsingSentenceExploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency
Independently trained machine learning models tend to learn similar features. Given an ensemble of independently trained models, this results in correlated predictions and common failure modes. Previous attempts focusing…
SAMa: Material-aware 3D Selection and Segmentation
Decomposing 3D assets into material parts is a common task for artists and creators, yet remains a highly manual process. In this work, we introduce Select Any Material (SAMa), a material selection approach for various 3…
Contrastive LearningText to 3DEnhancing Mathematical Reasoning in Large Language Models with Self-Consistency-Based Hallucination Detection
Large language models (LLMs) have demonstrated strong mathematical reasoning capabilities but remain susceptible to hallucinations producing plausible yet incorrect statements especially in theorem proving, symbolic mani…
Answer SelectionAutomated Theorem ProvingComputational EfficiencyHallucination+1