Adapter Pruning using Tropical Characterization
Adapters are widely popular parameter-efficient transfer learning approaches in natural language processing that insert trainable modules in between layers of a pre-trained language model. Apart from several heuristics, however, there has been a lack of studies analyzing the optimal number of adapter parameters needed for downstream applications. In this paper, we propose an adapter pruning approach by studying the tropical characteristics of trainable modules. We cast it as an optimization problem that aims to prune parameters from the adapter layers without changing the orientation of underlying tropical hypersurfaces. Our experiments on five NLP datasets show that tropical geometry tends to identify more relevant parameters to prune when compared with the magnitude-based baseline, while a combined approach works best across the tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On the Decision Boundaries of Deep Neural Networks: A Tropical Geometry Perspective
This work tackles the problem of characterizing and understanding the decision boundaries of neural networks with piece-wise linear non-linearity activations. We use tropical geometry, a new development in the area of al…
Network PruningOn the Decision Boundaries of Neural Networks. A Tropical Geometry Perspective
This work tackles the problem of characterizing and understanding the decision boundaries of neural networks with piecewise linear non-linearity activations. We use tropical geometry, a new development in the area of alg…
Network PruningOn the Decision Boundaries of Neural Networks: A Tropical Geometry Perspective
This work tackles the problem of characterizing and understanding the decision boundaries of neural networks with piecewise linear non-linearity activations. We use tropical geometry, a new development in the area of alg…
Network PruningTropical Geometrical Zonotope Reduction as Applied to Neural Network Compression.
Recently, tropical geometry had a significant impact on the study of deep neural networks. Motivated by this progress, we provide a novel framework for tropical polynomial approximation based on geometrical zonotope redu…
Neural Network CompressionStructured Pruning Adapters
Adapters are a parameter-efficient alternative to fine-tuning, which augment a frozen base network to learn new tasks. Yet, the inference of the adapted model is often slower than the corresponding fine-tuned model. To i…
Single Particle Analysis