Large Convolutional Model Tuning via Filter Subspace

2024-03-01 · Wei Chen, Zichen Miao, Qiang Qiu

Efficient fine-tuning methods are critical to address the high computational and parameter complexity while adapting large pre-trained models to downstream tasks. Our study is inspired by prior research that represents each convolution filter as a linear combination of a small set of filter subspace elements, referred to as filter atoms. In this paper, we propose to fine-tune pre-trained models by adjusting only filter atoms, which are responsible for spatial-only convolution, while preserving spatially-invariant channel combination knowledge in atom coefficients. In this way, we bring a new filter subspace view for model tuning. Furthermore, each filter atom can be recursively decomposed as a combination of another set of atoms, which naturally expands the number of tunable parameters in the filter subspace. By only adapting filter atoms constructed by a small number of parameters, while maintaining the rest of model parameters constant, the proposed approach is highly parameter-efficient. It effectively preserves the capabilities of pre-trained models and prevents overfitting to downstream tasks. Extensive experiments show that such a simple scheme surpasses previous tuning baselines for both discriminate and generative tasks.

📄 PDF Abstract BibTeX arXiv:2403.00269

Code (0)

등록된 구현이 없습니다.

Tasks

model

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models

2025-03-24 · CVPR 2025 1 · Zichen Miao, Wei Chen, Qiang Qiu

Transformer-based large pre-trained models have shown remarkable generalization ability, and various parameter-efficient fine-tuning (PEFT) methods have been proposed to customize these models on downstream tasks with mi…

parameter-efficient fine-tuning Tensor Decomposition

Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression

2018-03-15 · Dong Wang, Lei Zhou, Xueni Zhang, Xiao Bai 외

While the research on convolutional neural networks (CNNs) is progressing quickly, the real-world deployment of these models is often limited by computing resources and memory constraints. In this paper, we address this …

Clustering

A Joint Subspace View to Convolutional Neural Networks

2021-09-29 · Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, Qiang Qiu

Motivated by the intuition that important image regions remain important across different layers and scales in a CNN, we propose in this paper a joint subspace view to convolutional filters across network layers. When we…

Continual Learning with Filter Atom Swapping

2021-09-29 · ICLR 2022 4 · Zichen Miao, Ze Wang, Wei Chen, Qiang Qiu

Continual learning is widely studied in recent years to resolve the \textit{catastrophic forgetting} of deep neural networks. In this paper, we first enforce a low-rank filter subspace by decomposing convolutional filter…

Continual Learning

Graph Convolutional Networks via Adaptive Filter Banks

2021-09-29 · Xing Gao, Wenrui Dai, Chenglin Li, Junni Zou 외

Graph convolutional networks have been a powerful tool in representation learning of networked data. However, most architectures of message passing graph convolutional networks (MPGCNs) are limited as they employ a singl…

Representation Learning