CoLU

2000년 도입 · 논문 1편에서 사용

CoLU is an activation function similar to Swish and Mish in properties. It is defined as: $$f(x)=\frac{x}{1-x^{-(x+e^x)}}$$ It is smooth, continuously differentiable, unbounded above, bounded below, non-saturating, and non-monotonic. Based on experiments done with CoLU with different activation functions, it is observed that CoLU usually performs better than other functions on deeper neural networks.

출처: Deeper Learning with CoLU Activation

소개 논문: Deeper Learning with CoLU Activation