Activation Functions: Do They Represent A Trade-Off Between Modular Nature of Neural Networks And Task Performance
Current research suggests that the key factors in designing neural network architectures involve choosing number of filters for every convolution layer, number of hidden neurons for every fully connected layer, dropout and pruning. The default activation function in most cases is the ReLU, as it has empirically shown faster training convergence. We explore whether ReLU is the best choice if one is aiming to desire better modularity structure within a neural network.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Depth-Width Trade-offs for ReLU Networks via Sharkovsky's Theorem
Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet c…
Open-Ended Question AnsweringBalancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
Trainable activation functions, whose parameters are optimized alongside network weights, offer increased expressivity compared to fixed activation functions. Specifically, trainable activation functions defined as ratio…
Reinforcement LearningContinuous ControlContinual LearningGraph-Adaptive Activation Functions for Graph Neural Networks
Activation functions are crucial in graph neural networks (GNNs) as they allow defining a nonlinear family of functions to capture the relationship between the input graph data and their representations. This paper propo…
Recommendation SystemsFrequency and Generalisation of Periodic Activation Functions in Reinforcement Learning
Periodic activation functions, often referred to as learned Fourier features have been widely demonstrated to improve sample efficiency and stability in a variety of deep RL algorithms. Potentially incompatible hypothese…
reinforcement-learningGeometric Properties and Graph-Based Optimization of Neural Networks: Addressing Non-Linearity, Dimensionality, and Scalability
Deep learning models are often considered black boxes due to their complex hierarchical transformations. Identifying suitable architectures is crucial for maximizing predictive performance with limited data. Understandin…
Decision MakingTransfer Learning