Computational Design of Diverse Morphologies and Sensors for Vision & Robotics

Event: CVPR 2024 Tutorial · Duration: 184 min · ▶ Watch on YouTube

Abstract

This tutorial explores the computational design of diverse morphologies and sensors for vision and robotics. It delves into how intelligence is embodied not just in the brain but also in the physical and perceptual structures of agents, drawing inspiration from biological systems. The speakers present methodologies for automated design, including physics-based and learning-based approaches, to optimize robot bodies and visual sensors for specific tasks and environments. Key themes include specialization, darkness adaptation, resolution, and multimodality, highlighting the benefits of co-designing physical form and control for efficient and robust performance in cyberphysical systems.

Speakers

  • Amir Zamir — EPFL
  • Andrei Atanov — EPFL
  • Sönke Johnsen — Duke University
  • Andrew Spielberg — MIT / Harvard / Carnegie Mellon University

Talks (5)

  • 00:00:00 — Amir Zamir: Introduction
    • Introduces the tutorial’s theme by showcasing how intelligence is embedded not just in the brain but also in the body’s morphology, using examples like a dead fish swimming and diverse animal eyes, and outlines the schedule for visual and physical morphology discussions.
  • 01:32:00 — Andrei Atanov: Solving Vision Tasks with Simple Photoreceptors instead of Cameras & How to Design Them
    • Demonstrates that simple, low-resolution photoreceptors can effectively solve complex vision tasks like navigation and control, and introduces a computational design methodology to optimize their placement and parameters for specific tasks.
  • 03:10:00 — Sönke Johnsen: Measures and Models of Visual Acuity in Oceanic Animals
    • Explores the unique challenges of vision in oceanic environments, particularly in deep-sea conditions, and how animals adapt their visual systems, including eye size, acuity, and bioluminescence, to optimize perception in varying light and clarity.
  • 03:57:00 — Andrew Spielberg: Physical Morphology: Preliminaries and “Classical” Approaches
    • Explores the role of physical morphology in robotics, contrasting rigid and soft robot designs, and introduces classical methods like trajectory optimization and bilevel optimization for co-designing robot bodies and controllers.
  • 04:50:00 — Andrew Spielberg: Physical Morphology: Generative AI and Interfacing with the Real-World
    • Focuses on leveraging generative AI, specifically diffusion models, for automatic design of soft robots, emphasizing differentiable simulation, co-optimization of morphology and control, and the challenges of sim-to-real transfer and physical realization.

Key Takeaways

  • Intelligence is distributed across both brain (control) and body (morphology), and optimizing both simultaneously leads to more efficient and robust agents.
  • Automated design, leveraging differentiable simulation and generative AI, is crucial for exploring unintuitive design spaces and achieving specialization for diverse tasks and environments.
  • Simple photoreceptors, when optimally designed and placed, can achieve performance comparable to complex cameras for certain vision tasks, challenging traditional assumptions about sensor complexity.
  • Biological systems offer rich inspiration for morphological and perceptual adaptations, especially in extreme environments like the deep sea, highlighting the importance of ecological context and multimodality.
  • Co-designing physical morphology and control using joint optimization and differentiable simulation can significantly improve robot performance and enable the discovery of novel, effective designs, even for complex real-world applications.

Methods / Models / Datasets Mentioned

  • DiffPD
  • ChainQueen
  • Point-E
  • GLIDE
  • PPO
  • Adam
  • SQP
  • SNOPT
  • k-Means clustering
  • LSTM
  • ResNet50
  • ChatGPT
  • Robogrammar
  • Evolution Gym
  • EvoGym
  • MuJoCo
  • Gazebo
  • Bullet
  • Drake
  • JAX, M.D.
  • Taichi-Lang
  • NVIDIA Warp

Topics

Computational Design · Morphological Intelligence · Perceptual Morphology · Robotics · Vision Systems · Automated Design · Differentiable Simulation · Generative AI · Soft Robotics · Bio-inspiration · Co-design · Sensor Placement · Actuator Design

Notes

Open for commentary — connections to other work, critiques, follow-up reading.