InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360° Neural Radiance Fields

Event: CVPR 2024 · Duration: 5 min · ▶ Watch on YouTube

Abstract

This paper addresses the challenging problem of text-guided 3D-consistent object inpainting on 360° Neural Radiance Fields (NeRFs). The authors propose a novel method, InNeRF360, to overcome difficulties in consistently selecting objects across multiple viewpoints and filling removed regions plausibly in 3D. Their approach combines a multi-view consistent segmentation module with a 3D geometric prior to achieve artifact-free and perceptually consistent object removal and editing. The method demonstrates superior performance compared to prior work, producing high-quality inpainting results in complex 360° scenes.

Speakers

  • Dongqing Wang — IVRL, EPFL
  • Tong Zhang — IVRL, EPFL
  • Alaa Abboud — IVRL, EPFL
  • Sabine Süsstrunk — IVRL, EPFL

Talks (1)

  • 00:00:00 — Dongqing Wang: InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360° Neural Radiance Fields
    • A novel method for text-guided, 3D-consistent object inpainting and removal in 360-degree Neural Radiance Fields, addressing challenges in multi-view segmentation and 3D-plausible infilling.

Key Takeaways

  • InNeRF360 enables text-guided, 3D-consistent object inpainting and removal in 360° NeRF scenes.
  • The method introduces a multi-view consistent segmentation module that leverages depth information and SAM for accurate object selection across viewpoints.
  • A 3D geometric prior, trained on ShapeNet, is used to effectively remove floaters and fill missing regions with 3D consistency after 2D image inpainting.
  • The approach significantly reduces artifacts and improves background preservation compared to existing 2D and 3D inpainting methods.
  • InNeRF360 can also be extended to fine-grained object appearance editing in 3D scenes.

Methods / Models / Datasets Mentioned

  • SPIn-NeRF
  • Instruct-NeRFeNeREditing 3D Scenes with Instructions
  • SAM
  • Dino
  • ShapeNet
  • SPN-360
  • NeRFacto
  • In2n

Topics

Neural Radiance Fields (NeRF) · 360-degree scenes · Object inpainting · Text-guided editing · 3D consistency · Multi-view segmentation · Diffusion models · Geometric priors · Object removal

Notes

Open for commentary — connections to other work, critiques, follow-up reading.