CombU: A Combined Unit Activation for Fitting Mathematical Expressions with Neural Networks
The activation functions are fundamental to neural networks as they introduce non-linearity into data relationships, thereby enabling deep networks to approximate complex data relations. Existing efforts to enhance neural network performance have predominantly focused on developing new mathematical functions. However, we find that a well-designed combination of existing activation functions within a neural network can also achieve this objective. In this paper, we introduce the Combined Units activation (CombU), which employs different activation functions at various dimensions across different layers. This approach can be theoretically proven to fit most mathematical expressions accurately. The experiments conducted on four mathematical expression datasets, compared against six State-Of-The-Art (SOTA) activation function algorithms, demonstrate that CombU outperforms all SOTA algorithms in 10 out of 16 metrics and ranks in the top three for the remaining six metrics.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
ChemKANs for Combustion Chemistry Modeling and Acceleration
Efficient chemical kinetic model inference and application for combustion problems is challenging due to large ODE systems and wideley separated time scales. Machine learning techniques have been proposed to streamline t…
Inductive BiasKolmogorov-Arnold NetworksOvercoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
In past few years, linear rectified unit activation functions have shown its significance in the neural networks, surpassing the performance of sigmoid activations. RELU (Nair & Hinton, 2010), ELU (Clevert et al., 2015),…
DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning
Optimizing the combustion efficiency of a thermal power generating unit (TPGU) is a highly challenging and critical task in the energy industry. We develop a new data-driven AI system, namely DeepThermal, to optimize the…
Continuous ControlOffline RLreinforcement-learningReinforcement Learning (RL)Data Driven based Dynamic Correction Prediction Model for NOx Emission of Coal Fired Boiler
The real-time prediction of NOx emissions is of great significance for pollutant emission control and unit operation of coal-fired power plants. Aiming at dealing with the large time delay and strong nonlinear characteri…
feature selectionEvaluating Contrast Localizer for Identifying Causal Units in Social & Mathematical Tasks in Language Models
This work adapts a neuroscientific contrast localizer to pinpoint causally relevant units for Theory of Mind (ToM) and mathematical reasoning tasks in large language models (LLMs) and vision-language models (VLMs). Acros…
Mathematical Reasoning