Memorization in Deep Neural Networks: Does the Loss Function matter?
Deep Neural Networks, often owing to the overparameterization, are shown to be capable of exactly memorizing even randomly labelled data. Empirical studies have also shown that none of the standard regularization techniques mitigate such overfitting. We investigate whether the choice of the loss function can affect this memorization. We empirically show, with benchmark data sets MNIST and CIFAR-10, that a symmetric loss function, as opposed to either cross-entropy or squared error loss, results in significant improvement in the ability of the network to resist such overfitting. We then provide a formal definition for robustness to memorization and provide a theoretical explanation as to why the symmetric losses provide this robustness. Our results clearly bring out the role loss functions alone can play in this phenomenon of memorization.
Code (1)
Tasks
MemorizationSimilar Papers 제목 키워드 기반
Provable Separations between Memorization and Generalization in Diffusion Models
Diffusion models have achieved remarkable success across diverse domains, but they remain vulnerable to memorization -- reproducing training data rather than generating novel outputs. This not only limits their creative …
Large Loss Matters in Weakly Supervised Multi-Label Classification
Weakly supervised multi-label classification (WSML) task, which is to learn a multi-label classification using partially observed labels per image, is becoming increasingly important due to its huge annotation cost. In t…
ClassificationMemorizationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONUntrained neural networks can demonstrate memorization-independent abstract reasoning
The nature of abstract reasoning is a matter of debate. Modern artificial neural network (ANN) models, like large language models, demonstrate impressive success when tested on abstract reasoning problems. However, it ha…
MemorizationVisual ReasoningCapacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data
This paper studies how the model architecture and data configurations influence the empirical memorization capacity of generative transformers. The models are trained using synthetic text datasets derived from the System…
MemorizationBeyond Scaling Laws: Understanding Transformer Performance with Associative Memory
Increasing the size of a Transformer does not always lead to enhanced performance. This phenomenon cannot be explained by the empirical scaling laws. Furthermore, the model's enhanced performance is closely associated wi…
Memorization