What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics
Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder, we show that, under mild assumptions, the model can successfully memorize inliers while failing to memorize outliers during certain stages of early training. In particular, we characterize not only the emergence of the IM effect, but also its strength and persistence, and analyze how these properties depend on the data distribution and parameter initialization. In addition, building on these insights, we derive simple yet practical guidelines for enhancing the IM effect, including data preprocessing and parameter initialization schemes, achieving state-of-the-art performance on the ADBench datasets. Our findings provide a theoretical foundation for the IM effect and offer actionable directions for improving IM-based outlier detection methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Outlier DetectionSimilar Papers 제목 키워드 기반
Memorize Early, Then Query: Inlier-Memorization-Guided Active Outlier Detection
Outlier detection (OD) aims to identify abnormal instances, known as outliers or anomalies, by learning typical patterns of normal data, or inliers. Performing OD under an unsupervised regime-without any information abou…
Outlier DetectionActive LearningODIM: Outlier Detection via Likelihood of Under-Fitted Generative Models
The unsupervised outlier detection (UOD) problem refers to a task to identify inliers given training data which contain outliers as well as inliers, without any labeled information about inliers and outliers. It has been…
MemorizationOutlier DetectionALTBI: Constructing Improved Outlier Detection Models via Optimization of Inlier-Memorization Effect
Outlier detection (OD) is the task of identifying unusual observations (or outliers) from a given or upcoming data by learning unique patterns of normal observations (or inliers). Recently, a study introduced a powerful …
MemorizationOutlier DetectionPrivacy PreservingDeployment-Time Memorization in Foundation-Model Agents
Foundation-model agents are increasingly long-lived systems that remember users across interactions, making memorization an explicit deployment-time function rather than solely a property of model weights. Existing work …
Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning
Reinforcement learning has become a key paradigm for eliciting reasoning abilities in large language models, where exploration is crucial for discovering effective solution trajectories. Existing exploration methods typi…
Reinforcement Learning