Deep Unlearn: Benchmarking Machine Unlearning

2024-10-02 · Xavier F. Cadet, Anastasia Borovykh, Mohammad Malekzadeh, Sara Ahmadi-Abhari, Hamed Haddadi

Machine unlearning (MU) aims to remove the influence of particular data points from the learnable parameters of a trained machine learning model. This is a crucial capability in light of data privacy requirements, trustworthiness, and safety in deployed models. MU is particularly challenging for deep neural networks (DNNs), such as convolutional nets or vision transformers, as such DNNs tend to memorize a notable portion of their training dataset. Nevertheless, the community lacks a rigorous and multifaceted study that looks into the success of MU methods for DNNs. In this paper, we investigate 18 state-of-the-art MU methods across various benchmark datasets and models, with each evaluation conducted over 10 different initializations, a comprehensive evaluation involving MU over 100K models. We show that, with the proper hyperparameters, Masked Small Gradients (MSG) and Convolution Transpose (CT), consistently perform better in terms of model accuracy and run-time efficiency across different models, datasets, and initializations, assessed by population-based membership inference attacks (MIA) and per-sample unlearning likelihood ratio attacks (U-LiRA). Furthermore, our benchmark highlights the fact that comparing a MU method only with commonly used baselines, such as Gradient Ascent (GA) or Successive Random Relabeling (SRL), is inadequate, and we need better baselines like Negative Gradient Plus (NG+) with proper hyperparameter selection.

📄 PDF Abstract BibTeX arXiv:2410.01276

Code (0)

등록된 구현이 없습니다.

Tasks

BenchmarkingMachine Unlearning

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Benchmarking Federated Machine Unlearning methods for Tabular Data

2025-04-01 · Chenguang Xiao, Abhirup Ghosh, Han Wu, Shuo Wang 외

Machine unlearning, which enables a model to forget specific data upon request, is increasingly relevant in the era of privacy-centric machine learning, particularly within federated learning (FL) environments. This pape…

Benchmarking Computational Efficiency Federated Learning Machine Unlearning+1

Easy Data Unlearning Bench

2026-02-18 · Roy Rinberg, Pol Puigdemont, Martin Pawelczyk, Volkan Cevher arxiv

Evaluating machine unlearning methods remains technically challenging, with recent benchmarks requiring complex setups and significant engineering overhead. We introduce a unified and extensible benchmarking suite that s…

OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics

2025-06-14 · Vineeth Dorna, Anmol Mekala, Wenlong Zhao, Andrew McCallum 외

Robust unlearning is crucial for safely deploying large language models (LLMs) in environments where data privacy, model safety, and regulatory compliance must be ensured. Yet the task is inherently challenging, partly d…

Benchmarking

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

2026-07-23 · Lorenzo Orsingher, Thomas De Min, Massimiliano Mancini, Davide Talon 외 arxiv

Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations. To evaluate unlearning effectiveness in multimodal large language models (MLLMs), prior works …

Visual Question Answering

PISTOL: Dataset Compilation Pipeline for Structural Unlearning of LLMs

2024-06-24 · Xinchi Qiu, William F. Shen, Yihong Chen, Nicola Cancedda 외

Recently, machine unlearning, which seeks to erase specific data stored in the pre-trained or fine-tuned models, has emerged as a crucial protective measure for LLMs. However, unlearning approaches for LLMs that have bee…

Benchmarking Machine Unlearning