A Neural Transfer Function for a Smooth and Differentiable Transition Between Additive and Multiplicative Interactions
Existing approaches to combine both additive and multiplicative neural units either use a fixed assignment of operations or require discrete optimization to determine what function a neuron should perform. This leads either to an inefficient distribution of computational resources or an extensive increase in the computational complexity of the training procedure. We present a novel, parameterizable transfer function based on the mathematical concept of non-integer functional iteration that allows the operation each neuron performs to be smoothly and, most importantly, differentiablely adjusted between addition and multiplication. This allows the decision between addition and multiplication to be integrated into the standard backpropagation training procedure.
Code (1)
Similar Papers 제목 키워드 기반
A Differentiable Transition Between Additive and Multiplicative Neurons
Existing approaches to combine both additive and multiplicative neural units either use a fixed assignment of operations or require discrete optimization to determine what function a neuron should perform. However, this …
Safety-Critical Control of Discontinuous Systems with Nonsmooth Safe Sets
This paper studies the design of controllers for discontinuous dynamics that ensure the safety of non-smooth sets. The safe set is represented by arbitrarily nested unions and intersections of 0-superlevel sets of differ…
Closing the Motion Execution Gap: From Semantic Motion Task Constraints to Kinematic Control
This paper addresses the Motion Execution Gap, the disconnect between high-level symbolic task descriptions using semantic constraints and executable robot motions. Motion Statecharts are introduced as an executable symb…
Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation
We deal with the problem of gradient estimation for stochastic differentiable relaxations of algorithms, operators, simulators, and other non-differentiable functions. Stochastic smoothing conventionally perturbs the inp…
Pose EstimationSmooth Approximations of the Rounding Function
We propose novel smooth approximations to the classical rounding function, suitable for differentiable optimization and machine learning applications. Our constructions are based on two approaches: (1) localized sigmoid …