Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration
Deep neural networks have become foundational to advancements in multiple domains, including recommendation systems, natural language processing, and so on. Despite their successes, these models often contain incompatible parameters that can be underutilized or detrimental to model performance, particularly when faced with specific, varying data distributions. Existing research excels in removing such parameters or merging the outputs of multiple different pretrained models. However, the former focuses on efficiency rather than performance, while the latter requires several times more computing and storage resources to support inference. In this paper, we set the goal to explicitly improve these incompatible parameters by leveraging the complementary strengths of different models, thereby directly enhancing the models without any additional parameters. Specifically, we propose Compatibility-aware Knowledge Integration (CKI), which consists of Parameter Compatibility Assessment and Parameter Splicing, which are used to evaluate the knowledge content of multiple models and integrate the knowledge into one model, respectively. The integrated model can be used directly for inference or for further fine-tuning. We conduct extensive experiments on various datasets for recommendation and language tasks, and the results show that Compatibility-aware Knowledge Integration can effectively optimize incompatible parameters under multiple tasks and settings to break through the training limit of the original model without increasing the inference cost.
Code (0)
등록된 구현이 없습니다.
Tasks
Recommendation SystemsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evidential Trust-Aware Model Personalization in Decentralized Federated Learning for Wearable IoT
Decentralized federated learning (DFL) enables collaborative model training across edge devices without centralized coordination, offering resilience against single points of failure. However, statistical heterogeneity a…
Federated LearningCompatibility-Aware Dynamic Fine-Tuning for Large Language Models
Supervised Fine-Tuning (SFT) is the predominant paradigm for aligning large language models (LLMs), yet it suffers from optimization instability and limited generalization. Recent work attributes this issue to pathologic…
Reinforcement LearningVICTOR: Visual Incompatibility Detection with Transformers and Fashion-specific contrastive pre-training
For fashion outfits to be considered aesthetically pleasing, the garments that constitute them need to be compatible in terms of visual aspects, such as style, category and color. Previous works have defined visual compa…
Binary ClassificationRecommendation SystemsOptimal Acceptance of Incompatible Kidneys
Incompatibility between patient and donor is a major barrier in kidney transplantation (KT). The increasing shortage of kidney donors has driven the development of desensitization techniques to overcome this immunologica…
Improving Event Coreference Resolution by Learning Argument Compatibility from Unlabeled Data
Argument compatibility is a linguistic condition that is frequently incorporated into modern event coreference resolution systems. If two event mentions have incompatible arguments in any of the argument roles, they cann…
coreference-resolutionCoreference ResolutionEvent Coreference ResolutionTransfer Learning