Papers Re-basin
“Re-basin” 태그가 달린 논문 7편 · 필터 해제
A correlation-permutation approach for speech-music encoders model merging
Creating a unified speech and music model requires expensive pre-training. Model merging can instead create an unified audio model with minimal computational expense. However, direct merging is challenging when the model…
Re-basinUpdate Your Transformer to the Latest Release: Re-Basin of Task Vectors
Foundation models serve as the backbone for numerous specialized models developed through fine-tuning. However, when the underlying pretrained model is updated or retrained (e.g., on larger and more curated datasets), th…
Re-basinWolf2Pack: The AutoFusion Framework for Dynamic Parameter Fusion
In the rapidly evolving field of deep learning, specialized models have driven significant advancements in tasks such as computer vision and natural language processing. However, this specialization leads to a fragmented…
Multi-Task LearningRe-basinVanishing Feature: Diagnosing Model Merging and Beyond
Model merging offers an efficient way to combine pre-trained neural networks but often suffers from inconsistent performance, especially when merging models with different initializations. We identify the ``vanishing fea…
Linear Mode ConnectivitymodelRe-basinMode Combinability: Exploring Convex Combinations of Permutation Aligned Models
We explore element-wise convex combinations of two permutation-aligned neural network parameter vectors $\Theta_A$ and $\Theta_B$ of size $d$. We conduct extensive experiments by examining various distributions of such m…
Linear Mode ConnectivityRe-basinRe-basin via implicit Sinkhorn differentiation
The recent emergence of new algorithms for permuting models into functionally equivalent regions of the solution space has shed some light on the complexity of error surfaces, and some promising properties like mode conn…
Continual LearningIncremental LearningLinear Mode ConnectivityModels Alignment+1Git Re-Basin: Merging Models modulo Permutation Symmetries
The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms -- often variant…
Linear Mode ConnectivityRe-basin