ReGenAI Workshop CVPR 2024

Event: CVPR 2024 · Duration: 115 min · ▶ Watch on YouTube

Abstract

The ReGenAI Workshop at CVPR 2024 explored various dimensions of responsible generative AI, moving beyond mere performance metrics to address crucial societal implications. Discussions highlighted the need for democratizing AI through efficient, multi-stage approaches and recycling existing models for novel tasks. Speakers emphasized the importance of nuanced control, explainability, and transparency in generative AI systems, particularly in high-stakes domains. The workshop also delved into the challenges of defining and measuring “responsible AI” in practice, advocating for trustworthy measurements and robust evaluation methodologies for open-ended generative outputs. A key theme was the recognition of inherent biases in large language and vision models, underscoring the ongoing need to understand and mitigate these issues, even as new capabilities emerge.

Speakers

  • Björn Ommer — Ludwig Maximilian University of Munich
  • Besmira Nushi — Microsoft Research
  • Miranda Bogen — Center for Democracy & Technology
  • Ye Zhu — Rice University
  • Carolina Lopez — DTU

Talks (5)

  • 00:00:00 — Björn Ommer: Exploring Different Dimensions of Responsible GenAI
    • Discusses different facets of responsible GenAI, including democratization, efficient resource utilization, nuanced control, explainability, and open-sourcing critical parts, introducing Flow Matching and DepthFM.
  • 00:03:50 — Besmira Nushi: Revisiting Responsible AI for Generative Capabilities
    • Explores new challenges in Responsible AI related to increased risk surface, interactive scenarios, adversarial vs. benign usage, illusions of generality, and scaling evaluation for open-ended generative output.
  • 00:04:55 — Miranda Bogen: Responsible Generative AI - From the Ground Up
    • Addresses the challenges of defining and measuring ‘responsible AI’ in practice, emphasizing trustworthy measurements, understanding context, and the need to address foundational ‘unsolved problems’ from traditional ML.
  • 00:05:55 — Ye Zhu: GELDA: A generative language annotation framework to reveal visual biases in image generators
    • Introduces GELDA, a framework that uses LLMs for hierarchical attribute generation and VLMs for zero-shot annotation to analyze visual biases in image generators like Stable Diffusion XL.
  • 00:06:55 — Carolina Lopez: Latent Directions: A Simple Pathway to Bias Mitigation in Generative AI
    • Presents a method for bias mitigation in generative AI by identifying and manipulating ‘latent directions’ in the model’s embedding space, allowing for fine-grained and localized control over generated attributes.

Key Takeaways

  • Responsible GenAI requires focusing on dimensions beyond raw performance, including democratization, efficient resource utilization, nuanced control, explainability, and transparency.
  • Current evaluation methods for generative AI are often insufficient, lacking robust benchmarks for emerging capabilities and failing to account for contextual nuances, leading to potential ‘blind spots’ regarding biases and harms.
  • Integrating generative AI into real-world, high-stakes applications (e.g., law enforcement, healthcare) necessitates a deep understanding of contextual information and the potential for models to perpetuate or exacerbate existing societal biases and inequalities.
  • The community needs to develop dynamic, flexible frameworks for bias analysis and evaluation that can adapt to new domains, identify hidden biases, and provide actionable insights for mitigation, moving beyond static, predefined attribute lists.
  • Addressing the challenges of responsible GenAI requires a collaborative effort involving practitioners, policy makers, and researchers to define practical requirements, establish accountability, and ensure that the technology’s development aligns with societal values and ethical considerations.

Methods / Models / Datasets Mentioned

  • Stable Diffusion
  • ProlificDreamer
  • Emu
  • DepthFM
  • DreamPose
  • LDM
  • LCM-LoRA
  • CFM
  • SR3
  • ZoeDepth
  • Marigold
  • CLIP
  • DALL-E 2
  • DALL-E 3
  • GPT-4o
  • GPT-4
  • GPT-4-0613
  • Claude
  • BLS 2022
  • SD
  • GIS 2020
  • CLIP RN 50
  • ImageNet
  • MULTIMODAL CAUSAL TRACE
  • MULTEDIT
  • GELDA
  • Stable Diffusion XL
  • LLM
  • VLM
  • BLIP
  • OWLv2
  • GPT-3.5
  • MMMLI
  • GSM8K
  • MATH
  • HumanEval
  • Big-Bench Hard
  • DROP
  • HellaSwag
  • WinoGrande
  • ARC
  • Python coding tasks
  • GPT-4V
  • RAG
  • SIGMA
  • VISOR Score

Topics

Responsible AI · Generative AI · Bias Mitigation · Model Explainability · Fairness in AI · AI Governance · Measurement and Evaluation · Contextual Understanding · Open Source AI · Societal Impact of AI

Notes

Open for commentary — connections to other work, critiques, follow-up reading.