CVPR 2025 - 2nd Workshop on Neural Fields Beyond Conventional Cameras
Event: CVPR 2025 Workshop on Neural Fields Beyond Conventional Cameras · Duration: 190 min · ▶ Watch on YouTube
Abstract
This workshop session from CVPR 2025 explores the rapidly expanding field of Neural Fields Beyond Conventional Cameras. Speakers delve into the application of neural fields for various non-visual and challenging imaging modalities, including sonar, radar, and time-of-flight cameras, as well as their integration with physical constraints for scientific discovery in areas like medical imaging, non-line-of-sight imaging, and atomic-scale microscopy. Key themes include leveraging Gaussian Splatting for novel view synthesis and 3D reconstruction in diverse environments, addressing sensor-specific artifacts, and developing physically-grounded world simulators for autonomous systems. The session also highlights advancements in multi-modal neural fields, dynamic scene reconstruction from multi-epoch observations, and the development of memory-efficient and robust implicit neural representations for inverse problems, including methods to restore convexity in optimization.
Speakers
- Ilya Chugunov — Princeton
- Katie Skinner — Assistant Professor, University of Michigan
- Christian Richardt — Meta
- Jingyi Yu — ShanghaiTech University
- Youming Deng — Cornell University
- Sage Simhon — MIT
- Felix Heide — Princeton University
- Aviad Levis — University of Toronto
- Sara Fridovich-Keil — Stanford University
Talks (9)
- 00:00:00 — Ilya Chugunov: Welcome & Introduction
- Introduction to the 2nd Workshop on Neural Fields Beyond Conventional Cameras, highlighting the motivation for neural fields in various imaging modalities and outlining the day’s schedule.
- 00:03:43 — Katie Skinner: Splatting Beyond the Visible Spectrum: Gaussian Splatting for Radar, Sonar, and More
- Discusses applying Gaussian Splatting to non-visual modalities like sonar and radar for 3D reconstruction, novel view synthesis, and image restoration in challenging environments like underwater and autonomous driving.
- 00:29:50 — Christian Richardt: Time-of-Flight Neural Fields
- Explores the use of neural fields for time-of-flight (ToF) camera data, addressing challenges like low reflectivity, multiple reflections, and phase wrapping for dynamic scene reconstruction and novel view synthesis.
- 01:12:00 — Jingyi Yu: Neural Fields for All: Physics, World Models, and Beyond
- Presents a broad perspective on neural fields, integrating physical constraints for various applications including medical imaging (CT, MRI, ultrasound), non-line-of-sight imaging, and atomic-scale microscopy, emphasizing the importance of physics-informed models.
- 01:26:40 — Youming Deng: Self-Calibrating Gaussian Splatting for Large Field-of-View Reconstruction
- Introduces a self-calibrating Gaussian Splatting method for large field-of-view (FOV) reconstruction from distorted fisheye camera images, addressing challenges of traditional undistortion methods.
- 01:36:30 — Sage Simhon: Neural Refraction Fields for Image Verification
- Proposes using neural refraction fields and physical totems (glass spheres) to detect image manipulations by verifying consistency of light refraction patterns in the image.
- 01:46:09 — Felix Heide: Multi-modal Neural Fields for Robot Perception and Planning
- Discusses the use of multi-modal neural fields for robust robot perception and planning, particularly in autonomous trucking, by leveraging physically-grounded world simulators to generate diverse and challenging scenarios.
- 02:12:00 — Aviad Levis: Reconstructing the Cosmos with Physics Constrained Neural Fields
- Explores the application of physics-constrained neural fields to reconstruct astrophysical phenomena like protoplanetary disks and galactic black holes from sparse and noisy observational data, highlighting the importance of incorporating physical laws into neural field models.
- 02:38:00 — Sara Fridovich-Keil: Volume Representations for Inverse Problems
- Discusses volume representations for inverse problems, focusing on GA-Planes as a memory-efficient and expressive implicit neural representation, and exploring how convexity can be restored in optimization for improved stability and performance.
Key Takeaways
- Neural fields offer powerful tools for 3D reconstruction and novel view synthesis, extending beyond traditional RGB imaging to modalities like sonar, radar, and time-of-flight.
- Integrating physical models and constraints into neural fields is crucial for handling complex data, addressing sensor-specific artifacts, and achieving robust performance in challenging real-world and scientific applications.
- Generative world simulators built upon multi-modal neural fields can create diverse and challenging scenarios, enabling end-to-end training and validation for autonomous systems like self-driving trucks.
- Neural fields can be effectively applied to inverse problems in scientific discovery, allowing for the reconstruction of complex 3D and 4D structures from sparse, noisy, and multi-epoch observational data, even with uncertain underlying physics.
- Advancements in implicit neural representations, such as GA-Planes, focus on improving memory efficiency, expressiveness, and optimization stability, including strategies to restore convexity for better convergence guarantees.
Methods / Models / Datasets Mentioned
SonarSplatNeRFGaussian SplattingZSplatNeusisNeusis-NGPDSCAOneusTÖRRFRadar FieldsGated FieldsHS-NeRFF-TÖRRFKineTransVersePolynerMonerUSCTBentRay-NeRFNetTFGA-PlanesPlenoxelsDVGOTensoRFTri-PlanesK-PlanesMERFFFNSIRENWIREInstant-NGPGSplatBACONGridSVD
Topics
Neural Fields · Gaussian Splatting · Time-of-Flight Imaging · Multi-modal Perception · Physics-Constrained Neural Fields · 3D Reconstruction · Novel View Synthesis · Hyperspectral Imaging · Autonomous Driving · Inverse Problems
Notes
Open for commentary — connections to other work, critiques, follow-up reading.