Anti-DreamBooth: Protecting Users from Personalized Text-to-Image Synthesis

Event: ICCV23 PARIS · Duration: 5 min · ▶ Watch on YouTube

Abstract

This work introduces Anti-DreamBooth, a novel method designed to protect users from the misuse of personalized text-to-image synthesis models like DreamBooth. By adding subtle, imperceptible adversarial perturbations to user images before they are published, Anti-DreamBooth significantly degrades the quality of models trained on these protected images. The method employs Projected Gradient Descent (PGD) with an adaptive surrogate model, leveraging Alternating Surrogate and Perturbation Learning (ASPL) to effectively disrupt the fine-tuning process of powerful diffusion models. Extensive experiments demonstrate its effectiveness and robustness against various attack scenarios, including model mismatching, different noise budgets, image transformations, and other personalized techniques like Textual Inversion and LoRA.

Speakers

  • Thanh Van Le — VinAI Research, Hanoi, Vietnam
  • Hao Phung — VinAI Research, Hanoi, Vietnam
  • Thuan Hoang Nguyen — VinAI Research, Hanoi, Vietnam
  • Quan Dao — VinAI Research, Hanoi, Vietnam
  • Ngoc Tran — Vanderbilt University
  • Anh Tran — VinAI Research, Hanoi, Vietnam

Talks (1)

  • 00:00:00 — Presenter: Anti-DreamBooth: Protecting Users from Personalized Text-to-Image Synthesis
    • This talk introduces Anti-DreamBooth, a method to add imperceptible adversarial noise to images, preventing their misuse in personalized text-to-image synthesis models like DreamBooth by degrading the quality of generated outputs.

Key Takeaways

  • Anti-DreamBooth effectively protects user images from being used to train high-quality personalized text-to-image models like DreamBooth.
  • The proposed Alternating Surrogate and Perturbation Learning (ASPL) method is crucial for adapting the adversarial noise to the dynamic fine-tuning process of diffusion models.
  • Anti-DreamBooth demonstrates robustness against various real-world challenges, including different model versions, image transformations, and other personalization techniques.
  • The method successfully degrades both identity similarity and image quality in generated outputs, even when tested on real-world systems like Astria.ai.
  • The imperceptible adversarial noise added by Anti-DreamBooth significantly increases metrics like Face Detection Failure Rate (FDFR) and BRISQUE, while reducing Identity Similarity Matching (ISM) and SER-FQA.

Methods / Models / Datasets Mentioned

  • DreamBooth
  • Anti-DreamBooth
  • UnGANable
  • Glaze
  • Adversarial Example Does Good
  • Textual Inversion
  • LoRA DreamBooth
  • Projected Gradient Descent (PGD)
  • Fully-trained Surrogate Model Guidance (FSMG)
  • Alternating Surrogate and Perturbation Learning (ASPL)
  • VGGFace2
  • CelebA-HQ
  • ArcFace
  • SER-FIQ
  • BRISQUE
  • Stable Diffusion
  • Astria.ai

Topics

Personalized text-to-image synthesis · DreamBooth · Adversarial attacks · Image protection · Diffusion models · Fine-tuning disruption · Privacy · Generative AI · Robustness

Notes

Open for commentary — connections to other work, critiques, follow-up reading.