AdaCap: An Adaptive Contrastive Approach for Small-Data Neural Networks
Neural networks struggle on small tabular datasets, where tree-based models remain dominant. We introduce Adaptive Contrastive Approach (AdaCap), a training scheme that combines a permutation-based contrastive loss with a Tikhonov-based closed-form output mapping. Across 85 real-world regression datasets and multiple architectures, AdaCap yields consistent and statistically significant improvements in the small-sample regime, particularly for residual models. A meta-predictor trained on dataset characteristics (size, skewness, noise) accurately anticipates when AdaCap is beneficial. These results show that AdaCap acts as a targeted regularization mechanism, strengthening neural networks precisely where they are most fragile. All results and code are publicly available at https://github.com/BrunoBelucci/adacap.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
AdaCap: Adaptive Capacity control for Feed-Forward Neural Networks
The capacity of a ML model refers to the range of functions this model can approximate. It impacts both the complexity of the patterns a model can learn but also memorization, the ability of a model to fit arbitrary labe…
MemorizationTADACap: Time-series Adaptive Domain-Aware Captioning
While image captioning has gained significant attention, the potential of captioning time-series images, prevalent in areas like finance and healthcare, remains largely untapped. Existing time-series captioning methods t…
Image CaptioningRetrievalTime SeriesAdaptive Multi-User Channel Estimation Based on Contrastive Feature Learning
Correlation exploitation is essential for efficient multi-user channel estimation (MUCE) in massive MIMO systems. However, the existing works either rely on presumed strong correlation or learn the correlation through la…
Contrastive LearningAdversarial Contrastive Distillation with Adaptive Denoising
Adversarial Robustness Distillation (ARD) is a novel method to boost the robustness of small models. Unlike general adversarial training, its robust knowledge transfer can be less easily restricted by the model capacity.…
Adversarial RobustnessDenoisingTransfer LearningAdversarial Lagrangian Integrated Contrastive Embedding for Limited Size Datasets
Certain datasets contain a limited number of samples with highly various styles and complex structures. This study presents a novel adversarial Lagrangian integrated contrastive embedding (ALICE) method for small-sized d…