Neuromodulation Gated Transformer
We introduce a novel architecture, the Neuromodulation Gated Transformer (NGT), which is a simple implementation of neuromodulation in transformers via a multiplicative effect. We compare it to baselines and show that it results in the best average performance on the SuperGLUE benchmark validation sets.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Neuromodulation and homeostasis: complementary mechanisms for robust neural function
Neurons depend on two interdependent mechanisms-homeostasis and neuromodulation-to maintain robust and adaptable functionality. Homeostasis stabilizes neuronal activity by adjusting ionic conductances, whereas neuromodul…
Electronic schematic for bio-plausible dopamine neuromodulation of eSTDP and iSTDP
In this technical report we present novel results of the dopamine bio-plausible neuromodulation excitatory (eSTDP) and inhibitory (iSTDP) learning. We present the principal schematic for the neuromodulation of D1 and D2 …
Cellular neuromodulation in artificial networks
Animals excel at adapting their intentions, attention, and actions to the environment, making them remarkably efficient at interacting with a rich, unpredictable and ever-changing external world, a property that intellig…
Meta-LearningIntroducing Neuromodulation in Deep Neural Networks to Learn Adaptive Behaviours
Animals excel at adapting their intentions, attention, and actions to the environment, making them remarkably efficient at interacting with a rich, unpredictable and ever-changing external world, a property that intellig…
Meta Reinforcement LearningReinforcement LearningNeuromodulated Learning in Deep Neural Networks
In the brain, learning signals change over time and synaptic location, and are applied based on the learning history at the synapse, in the complex process of neuromodulation. Learning in artificial neural networks, on t…