The missing rungs on the ladder to general AI

Event: ICCV 2023 VLAR workshop · Duration: 49 min · ▶ Watch on YouTube

Abstract

This workshop session by Francois Chollet discusses the current limitations of large language models (LLMs) in achieving Artificial General Intelligence (AGI), despite their impressive performance on benchmarks. He argues that LLMs primarily excel at ‘type 1 abstraction’ (pattern matching and interpolation) but lack ‘type 2 abstraction’ (true reasoning and generalization to unfamiliar problems). The presentation highlights issues like data contamination, hallucinations, sensitivity to phrasing, and weak generalization in LLMs, proposing that measuring and maximizing generalization, particularly through program-centric abstraction and discrete program search, is crucial for progress towards AGI. The Abstraction and Reasoning Corpus (ARC) is introduced as a benchmark designed to test these higher-level generalization abilities.

Speakers

  • Francois Chollet — Google, Inc.

Talks (1)

  • 00:00:00 — Francois Chollet: The missing rungs on the ladder to general AI
    • An overview of the current state of AI, particularly LLMs, highlighting their limitations in achieving true generalization and proposing abstraction as the key to progress towards AGI.

Key Takeaways

  • Current LLMs, despite high benchmark scores, exhibit significant limitations in true generalization, often due to data contamination and reliance on memorized patterns.
  • The core challenge for AGI lies in achieving ‘type 2 abstraction,’ which involves reasoning and adapting to novel, unfamiliar situations, rather than just ‘type 1 abstraction’ (pattern matching).
  • Measuring and maximizing generalization, controlling for experience and priors, is essential for making progress towards AGI.
  • Program-centric abstraction, involving the reuse and synthesis of programs, offers a promising path to developing more robust and generalizable AI systems.
  • Merging deep learning for perception with discrete program search for reasoning could be a key strategy for bridging the gap towards AGI.

Methods / Models / Datasets Mentioned

  • GPT-4
  • Bard
  • GPT-3.5
  • RLHF
  • Transformers
  • Word2vec
  • Hebbian learning
  • Self-attention
  • Hypothesis Search: Inductive Reasoning with Language Models

Topics

Artificial General Intelligence (AGI) · Large Language Models (LLMs) · Generalization · Abstraction (Type 1 and Type 2) · Data Contamination · Hallucinations · Program Synthesis · Discrete Program Search · Fluid Intelligence · Core Knowledge · Abstraction and Reasoning Corpus (ARC)

Notes

Open for commentary — connections to other work, critiques, follow-up reading.