Papers Linear Mode Connectivity
“Linear Mode Connectivity” 태그가 달린 논문 35편 · 필터 해제
Understanding Mode Connectivity via Parameter Space Symmetry
Neural network minima are often connected by curves along which train and test loss remain nearly constant, a phenomenon known as mode connectivity. While this property has enabled applications such as model merging and …
Linear Mode ConnectivityCodeMerge: Codebook-Guided Model Merging for Robust Test-Time Adaptation in Autonomous Driving
Maintaining robust 3D perception under dynamic and unpredictable test-time conditions remains a critical challenge for autonomous driving systems. Existing test-time adaptation (TTA) methods often fail in high-variance t…
3D Object DetectionAutonomous DrivingLinear Mode Connectivitymotion prediction+3Model Assembly Learning with Heterogeneous Layer Weight Merging
Model merging acquires general capabilities without extra data or training by combining multiple models' parameters. Previous approaches achieve linear mode connectivity by aligning parameters into the same loss basin us…
Linear Mode ConnectivityFinding Stable Subnetworks at Initialization with Dataset Distillation
Recent works have shown that Dataset Distillation, the process for summarizing the training data, can be leveraged to accelerate the training of deep learning models. However, its impact on training dynamics, particularl…
Dataset DistillationLinear Mode ConnectivityNetwork PruningAnalyzing the Role of Permutation Invariance in Linear Mode Connectivity
It was empirically observed in Entezari et al. (2021) that when accounting for the permutation invariance of neural networks, there is likely no loss barrier along the linear interpolation between two SGD solutions -- a …
Linear Mode ConnectivityThe Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE
Recent studies have shown that reducing symmetries in neural networks enhances linear mode connectivity between networks without requiring parameter space alignment, leading to improved performance in linearly interpolat…
Linear Mode ConnectivityMixture-of-ExpertsDeep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond
Deep learning sometimes appears to work in unexpected ways. In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network consisti…
Linear Mode ConnectivityCopRA: A Progressive LoRA Training Strategy
Low-Rank Adaptation (LoRA) is a parameter-efficient technique for rapidly fine-tuning foundation models. In standard LoRA training dynamics, models tend to quickly converge to a local optimum near the initialization. How…
Federated LearningLinear Mode ConnectivityMulti-Task LearningThe Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse
Model merging aims to efficiently combine the weights of multiple expert models, each trained on a specific task, into a single multi-task model, with strong performance across all tasks. When applied to all but the last…
Linear Mode ConnectivityTask ArithmeticApproaching Deep Learning through the Spectral Dynamics of Weights
We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phenomena in deep learning. We identify a c…
Deep Learningimage-classificationImage ClassificationImage Generation+5Landscaping Linear Mode Connectivity
The presence of linear paths in parameter space between two different network solutions in certain cases, i.e., linear mode connectivity (LMC), has garnered interest from both theoretical and practical fronts. There has …
Linear Mode ConnectivityUnveiling the Dynamics of Information Interplay in Supervised Learning
In this paper, we use matrix information theory as an analytical tool to analyze the dynamics of the information interplay between data representations and classification head vectors in the supervised learning process. …
Linear Mode ConnectivityThe Empirical Impact of Neural Parameter Symmetries, or Lack Thereof
Many algorithms and observed phenomena in deep learning appear to be affected by parameter symmetries -- transformations of neural network parameters that do not change the underlying neural network function. These inclu…
Linear Mode ConnectivityLinear Mode Connectivity in Differentiable Tree Ensembles
Linear Mode Connectivity (LMC) refers to the phenomenon that performance remains consistent for linearly interpolated models in the parameter space. For independently optimized model pairs from different random initializ…
Linear Mode ConnectivityFederated Learning over Connected Modes
Statistical heterogeneity in federated learning poses two major challenges: slow global training due to conflicting gradient signals, and the need of personalization for local distributions. In this work, we tackle both …
ClusteringFederated LearningLinear Mode ConnectivityImproving Group Connectivity for Generalization of Federated Deep Learning
Federated learning (FL) involves multiple heterogeneous clients collaboratively training a global model via iterative local updates and model fusion. The generalization of FL's global model has a large gap compared with …
Deep LearningFederated LearningLinear Mode ConnectivityAnalysis of Linear Mode Connectivity via Permutation-Based Weight Matching
Recently, Ainsworth et al. showed that using weight matching (WM) to minimize the $L_2$ distance in a permutation search of model parameters effectively identifies permutations that satisfy linear mode connectivity (LMC)…
Linear Mode ConnectivityVanishing Feature: Diagnosing Model Merging and Beyond
Model merging offers an efficient way to combine pre-trained neural networks but often suffers from inconsistent performance, especially when merging models with different initializations. We identify the ``vanishing fea…
Linear Mode ConnectivitymodelRe-basinTraining-time Neuron Alignment through Permutation Subspace for Improving Linear Mode Connectivity and Model Fusion
In deep learning, stochastic gradient descent often yields functionally similar yet widely scattered solutions in the weight space even under the same initialization, causing barriers in the Linear Mode Connectivity (LMC…
Federated LearningLinear Mode ConnectivityDisentangling Linear Mode-Connectivity
Linear mode-connectivity (LMC) (or lack thereof) is one of the intriguing characteristics of neural network loss landscapes. While empirically well established, it unfortunately still lacks a proper theoretical understan…
Linear Mode Connectivity