Are All Linear Regions Created Equal?
The number of linear regions has been studied as a proxy of complexity for ReLU networks. However, the empirical success of network compression techniques like pruning and knowledge distillation, suggest that in the overparameterized setting, linear regions density might fail to capture the effective nonlinearity. In this work, we propose an efficient algorithm for discovering linear regions and use it to investigate the effectiveness of density in capturing the nonlinearity of trained VGGs and ResNets on CIFAR-10 and CIFAR-100. We contrast the results with a more principled nonlinearity measure based on function variation, highlighting the shortcomings of linear regions density. Furthermore, interestingly, our measure of nonlinearity clearly correlates with model-wise deep double descent, connecting reduced test error with reduced nonlinearity, and increased local similarity of linear regions.
Code (1)
Tasks
AllKnowledge DistillationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance
Supervised fine-tuning (SFT) is a pivotal approach to adapting large language models (LLMs) for downstream tasks; however, performance often suffers from the ``seesaw phenomenon'', where indiscriminate parameter updates …
Not All Character N-grams Are Created Equal: A Study in Authorship Attribution
Not All Titles are Created Equal: Financial Document Structure Extraction Shared Task
Not All Contexts Are Created Equal: Better Word Representations with Variable Attention
Maximum Likelihood Identification of Linear Models with Integrating Disturbances for Offset-Free Control
This paper addresses the identification of models for offset-free model predictive control (MPC), where LTI models are augmented with (fictitious) uncontrollable integrating modes, called integrating disturbances. The st…
Model Predictive Control