홈 › Papers

Quantile Activation: Correcting a Failure Mode of ML Models

An established failure mode for machine learning models occurs when the same features are equally likely to belong to class 0 and class 1. In such cases, existing ML models cannot correctly classify the sample. However, a solvable case emerges when the probabilities of class 0 and 1 vary with the context distribution. To the best of our knowledge, standard neural network architectures like MLPs or CNNs are not equipped to handle this. In this article, we propose a simple activation function, quantile activation (QACT), that addresses this problem without significantly increasing computational costs. The core idea is to adapt the outputs of each neuron to its context distribution. The proposed quantile activation, QACT, produces the relative quantile of the sample in its context distribution, rather than the actual values, as in traditional networks. A practical example where the same sample can have different labels arises in cases of inherent distribution shift. We validate the proposed activation function under such shifts, using datasets designed to test robustness against distortions : CIFAR10C, CIFAR100C, MNISTC, TinyImagenetC. Our results demonstrate significantly better generalization across distortions compared to conventional classifiers, across various architectures. Although this paper presents a proof of concept, we find that this approach unexpectedly outperforms DINOv2 (small) under large distortions, despite DINOv2 being trained with a much larger network and dataset.

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반