Scenic: An Open-Source Probabilistic Programming System for Data Generation and Safety in AI-Based Autonomy

Event: CVPR 2024 Tutorial · Duration: 133 min · ▶ Watch on YouTube

Abstract

This tutorial introduces Scenic, an open-source probabilistic programming system designed for data generation and ensuring safety in AI-based autonomous systems. It covers the core concepts of Scenic and its companion tool, VerifAI, demonstrating their application through industrial case studies in autonomous airplane taxiing and self-driving cars. Participants gain hands-on experience with the Scenic 3.0 language, learning to define static and dynamic scenarios, model complex behaviors, and generate synthetic data. The tutorial also explores advanced applications such as debugging machine learning models, validating simulations against real-world data, querying large sensor datasets, and training human motor skills in extended reality environments. The overarching goal is to provide a comprehensive understanding of how Scenic and VerifAI contribute to the development and verification of safe and robust AI systems.

Speakers

  • Sanjit A. Seshia — UC Berkeley
  • Daniel J. Fremont — UC Santa Cruz
  • Edward Kim — UC Berkeley

Talks (7)

  • 00:00:00 — Sanjit A. Seshia: Introduction to Scenic and VerifAI: Context and Overview
    • Introduces the Scenic project as an open-source probabilistic programming system for data generation and AI safety, highlighting its evolution and broad applications in autonomous systems.
  • 01:34:00 — Sanjit A. Seshia: Industrial Case Study 1: Autonomous Airplane Taxiing
    • Presents a case study on autonomous airplane taxiing (TaxiNet) using Scenic and VerifAI, demonstrating how to model scenarios, falsify safety properties, debug failures, retrain models, and implement runtime assurance.
  • 02:52:00 — Sanjit A. Seshia: Industrial Case Study 2: Autonomous Driving (Sim-to-Real Validation)
    • Explores a sim-to-real validation case study for autonomous driving involving vulnerable road users, showcasing how Scenic and VerifAI can be used to generate test cases, conduct real-world experiments, and identify perception failures.
  • 03:46:00 — Daniel J. Fremont: Hands-on Introduction to the Scenic 3.0 Language
    • Provides a hands-on introduction to the Scenic 3.0 language, covering static and dynamic scenario definitions, probabilistic programming features, behaviors, actions, and advanced temporal constructs, with practical Colab notebook exercises.
  • 05:00:00 — Edward Kim: Applications of Scenic (Part I): Sensor Data Collection and Testing ML Components
    • Details how Scenic facilitates sensor data collection and generation for ML models, and how VerifAI enables testing of perception, behavior prediction, and planning components by intelligently sampling scenarios from a defined distribution.
  • 05:54:00 — Edward Kim: Applications of Scenic (Part II): Debugging, Sim-to-Real, Data Exploration, and XR
    • Covers advanced applications including debugging ML models by extracting failure-inducing rules, validating simulations against real-world scenarios, exploring large sensor datasets using Scenic as a query language, and training human motor skills in extended reality environments.
  • 06:45:00 — Sanjit A. Seshia: Summary and Q&A
    • Summarizes the tutorial’s key topics and applications of Scenic and VerifAI, emphasizing their roles in data generation, testing, debugging, and sim-to-real validation, followed by a Q&A session.

Key Takeaways

  • Scenic is a high-level probabilistic programming language that allows modular and compositional definition of complex scenarios, including probabilistic parameters and dynamic agent behaviors.
  • VerifAI, a supporting toolkit, enables systematic testing, falsification, debugging, and synthesis of AI-based systems by intelligently sampling scenarios and analyzing performance metrics.
  • Scenic and VerifAI facilitate sim-to-real validation by generating diverse test cases in simulation that can be replicated in the real world, helping to identify and address discrepancies.
  • The framework supports flexible sensor data generation, allowing users to specify various sensor types and logging rates, and can be used as a query language for exploring and retrieving specific data from large datasets.
  • New developments include generating Scenic programs from natural language descriptions of crash reports and applying Scenic to train human motor skills in extended reality environments.

Methods / Models / Datasets Mentioned

  • VerifAI
  • TaxiNet
  • Apollo 3.5
  • SMT (Satisfiability Modulo Theory)
  • NuScenes
  • Bayesian Knowledge Tracing (BKT)

Topics

Probabilistic Programming · Data Generation · AI Safety · Autonomous Systems · Simulation · Verification · Debugging ML Models · Sim-to-Real Validation · Sensor Data Exploration · Extended Reality

Notes

Open for commentary — connections to other work, critiques, follow-up reading.