Interpolating Compressed Parameter Subspaces
Inspired by recent work on neural subspaces and mode connectivity, we revisit parameter subspace sampling for shifted and/or interpolatable input distributions (instead of a single, unshifted distribution). We enforce a compressed geometric structure upon a set of trained parameters mapped to a set of train-time distributions, denoting the resulting subspaces as Compressed Parameter Subspaces (CPS). We show the success and failure modes of the types of shifted distributions whose optimal parameters reside in the CPS. We find that ensembling point-estimates within a CPS can yield a high average accuracy across a range of test-time distributions, including backdoor, adversarial, permutation, stylization and rotation perturbations. We also find that the CPS can contain low-loss point-estimates for various task shifts (albeit interpolated, perturbed, unseen or non-identical coarse labels). We further demonstrate this property in a continual learning setting with CIFAR100.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningSimilar Papers 제목 키워드 기반
AdaBoost and robust one-bit compressed sensing
This paper studies binary classification in robust one-bit compressed sensing with adversarial errors. It is assumed that the model is overparameterized and that the parameter of interest is effectively sparse. AdaBoost …
Binary Classificationcompressed sensingGeneral ClassificationRestricted Isometry Property of Gaussian Random Projection for Finite Set of Subspaces
Dimension reduction plays an essential role when decreasing the complexity of solving large-scale problems. The well-known Johnson-Lindenstrauss (JL) Lemma and Restricted Isometry Property (RIP) admit the use of random p…
Clusteringcompressed sensingDimensionality ReductionLEMMACompress Then Adapt? No, Do It Together via Task-aware Union of Subspaces
Adapting large pretrained models to diverse tasks is now routine, yet the two dominant strategies of parameter-efficient fine-tuning (PEFT) and low-rank compression are typically composed in sequence. This decoupled prac…
parameter-efficient fine-tuningCompressed Subspace Learning Based on Canonical Angle Preserving Property
Union of Subspaces (UoS) is a popular model to describe the underlying low-dimensional structure of data. The fine details of UoS structure can be described in terms of canonical angles (also known as principal angles) b…
ClusteringDimensionality ReductionSpatial Sparse subspace clustering for Compressive Spectral imaging
This paper aims at developing a clustering approach with spectral images directly from CASSI compressive measurements. The proposed clustering method first assumes that compressed measurements lie in the union of multipl…
ClusteringImage Clustering