Machine Unlearning in Computer Vision: Foundations and Applications

Event: CVPR 2024 Tutorial · Duration: 197 min · ▶ Watch on YouTube

Abstract

This tutorial provides a comprehensive overview of Machine Unlearning (MU) in Computer Vision, covering its foundational concepts, motivations, and practical applications. It delves into various algorithmic approaches, including influence unlearning and gradient-based methods, and explores model-centric strategies like sparsity and saliency for efficient unlearning. The tutorial also addresses the critical aspect of evaluating unlearning performance, introduces the concept of “worst-case unlearning” to understand limitations, and discusses real-world applications in regulated environments, privacy-aware systems, and the removal of problematic data. Finally, it highlights open research questions and challenges in designing and evaluating robust MU methods across different modalities and problem formulations.

Speakers

  • Sijia Liu — Michigan State University
  • Yang Liu — UC Santa Cruz
  • Nathalie Baracaldo — IBM Research
  • Eleni Triantafillou — Google

Talks (8)

  • 00:00:00 — Sijia Liu: Machine Unlearning in Computer Vision: Foundations and Applications
    • Introduces the concept of machine unlearning, its motivation (data privacy, backdoor defense, transfer learning improvement, concept erasing in generative models), and outlines the tutorial’s agenda.
  • 00:02:40 — Sijia Liu: Part 1 Introduction
    • Defines Machine Unlearning (MU) as erasing the influence of undesirable data/classes in model performance without costly retraining, highlighting its applications beyond privacy like Trojan AI defense and improved transfer learning.
  • 00:05:37 — Eleni Triantafillou: Part 2 Evaluating Unlearning: Progress and Challenges
    • Discusses the formal definition of unlearning, the NeurIPS unlearning competition, and the rigorous evaluation framework developed, emphasizing the need to measure how close an unlearned model is to a retrained model.
  • 00:08:49 — Yang Liu: Part 3 Algorithmic Foundations of MU
    • Reviews various machine unlearning methods, focusing on Influence Unlearning (IU) and Gradient Ascent (GA) approaches, and delves into the theoretical underpinnings of IU and why GA works but can also fail.
  • 00:11:30 — Sijia Liu: Part 4 Exploring and Exploiting Model Characteristics for Ease of Unlearning
    • Explores model-centric approaches to unlearning, focusing on exploiting weight sparsity and weight saliency (gradient-based) to facilitate unlearning, particularly for discriminative and generative models.
  • 00:15:50 — Sijia Liu: Part 5 When will be the worst case for unlearning?
    • Investigates the concept of “worst-case unlearning” from a data selection perspective, proposing a bi-level optimization framework to identify forget sets that are maximally difficult to unlearn, and demonstrates its application in image classification and text-to-image generation.
  • 00:20:25 — Nathalie Baracaldo: Part VI Let’s make it practical: Unlearning Application with Real Requirements
    • Discusses practical applications of unlearning in industry, focusing on regulated environments (exact unlearning, bias), privacy-aware systems (federated learning), and removing problematic data (backdoors, toxic content), highlighting the challenges and proposed methods for each.
  • 00:25:50 — Eleni Triantafillou: Conclusions and open questions
    • Summarizes the tutorial’s content, highlights key takeaways regarding different problem formulations, modalities, and the interpretability of unlearning difficulty, and poses open questions for future research in evaluation and algorithm design.

Key Takeaways

  • Machine Unlearning (MU) is crucial for addressing data privacy, security, and ethical concerns in AI models by efficiently removing the influence of specific data without costly retraining.
  • Various algorithmic approaches exist, ranging from influence-based methods that estimate parameter changes to gradient-based techniques that directly manipulate model weights.
  • Evaluating MU requires a rigorous framework that considers computational efficiency, preserved model utility, and unlearning efficacy, often involving adversarial attacks to assess true forgetting.
  • Exploiting model characteristics like sparsity and saliency can significantly improve the efficiency and effectiveness of unlearning, particularly in complex deep learning architectures.
  • Unlearning extends beyond simple data deletion, finding applications in mitigating backdoor attacks, improving transfer learning, and enabling concept erasing in generative models, with emerging challenges in large and multimodal systems.

Methods / Models / Datasets Mentioned

  • Fine-tune (FT)
  • Gradient Ascent (GA)
  • Influence Unlearning (IU)
  • Fisher Forgetting (FF)
  • One-Shot Magnitude Pruning (OMP)
  • Label Smoothing (LS)
  • NegGrad+
  • NegGrad
  • SalUn
  • Random-label
  • SPUNGE
  • RMU (Representation Misdirection Unlearning)
  • Erased Stable Diffusion (ESD)
  • Forget-Me-Not (FMN)
  • Safe Latent Diffusion (SLD)
  • P4D (Prompting 4 Debugging)
  • UnlearnDiffAtk
  • SISA (Sharded, Isolated, Sliced, and Aggregated)
  • FairSISA
  • HateXplain
  • Prompt Transformer
  • Embedding Corrupted Prompts (ECO)
  • Projected Gradient Ascent

Topics

Machine Unlearning (MU) definition · Data privacy and regulations · Backdoor defense · Transfer learning improvement · Concept erasing in generative models · Unlearning evaluation metrics · Algorithmic foundations of MU · Influence Unlearning (IU) · Gradient Ascent (GA) · Model sparsity and saliency · Worst-case unlearning · Federated learning · Adversarial attacks on unlearning · Interpretability of unlearning · Unlearning in large language models (LLMs)

Notes

Open for commentary — connections to other work, critiques, follow-up reading.