I/O 2024: Android

Year: 2024 · ▶ 在 YouTube 观看

Sameer Samat (President, Android Ecosystem) · Dave Burke (VP of Engineering, Android)

Switch language → en

话题段落 (4)

  • 00:00:12 · Introduction: Android with AI at the Core — Sameer Samat
    • Introduces the vision for reimagining the Android experience by deeply integrating AI, focusing on three key breakthroughs for the year.
  • 00:01:09 · Circle to Search Updates — Sameer Samat
    • Demonstrates new capabilities for Circle to Search, including solving complex homework problems with step-by-step instructions.
  • 00:03:57 · Gemini on Android: A Context-Aware Assistant — Dave Burke
    • Introduces Gemini as a system-level, context-aware AI assistant that works as an overlay on top of any app.
  • 00:08:19 · On-Device AI: Gemini Nano with Multimodality — Dave Burke
    • Explains how on-device foundation models like Gemini Nano will enable faster, private, and more capable AI experiences, including accessibility and scam protection.

产品发布 (6)

  • 00:02:15 · Circle to Search (Homework Help) (New Feature)
    • Solves math and physics word problems · Provides step-by-step instructions · Works directly on the screen where the problem is displayed
    • Available today
  • 00:04:08 · Gemini on Android (Overlay) (Enhanced Feature)
    • Works as a system-level overlay on any app · Context-aware suggestions based on on-screen content (e.g., ‘Ask this video’, ‘Ask this PDF’) · Drag-and-drop generated content into other apps
    • Rolling out over the next couple of months
  • 00:08:45 · Gemini Nano with Multimodality (Coming Soon)
    • On-device processing of text, images, and audio · Enables faster, private AI experiences · Powers features like enhanced TalkBack and scam detection
    • Coming to Pixel later this year
  • 00:09:06 · TalkBack with Gemini Nano (Upcoming Update)
    • Provides richer, more detailed descriptions of unlabeled images · Processes information on-device for speed and privacy · Works offline
    • Coming later this year
  • 00:10:04 · On-device Fraud Detection (In Testing)
    • Monitors phone calls in real-time for scam patterns · Provides an alert if a likely scam is detected · All audio processing is done on-device
    • Updates later this summer
  • 00:12:19 · Android 15 Beta 2 (Beta)
    • Coming tomorrow

时间承诺 (7)

  • 00:03:11 (Today) — Circle to Search homework help is available today.
  • 00:03:14 (Later this year) — Circle to Search will be able to solve more complex problems involving symbolic formulas, diagrams, and graphs.
  • 00:03:32 (By the end of 2024) — Circle to Search will be available on over 200 million devices.
  • 00:08:16 (Tomorrow) — Android 15 Beta 2 will be released.
  • 00:08:45 (Later this year) — Gemini Nano with Multimodality is coming to Pixel.
  • 00:09:57 (Later this year) — Updates to TalkBack powered by Gemini Nano are coming.
  • 00:11:24 (Later this summer) — More updates on the on-device fraud detection feature will be shared.

演示 (4)

  • 00:02:30 ✓ · Circle to Search for Homework — Sameer Samat
    • A user circles a physics word problem on their phone, and Circle to Search provides a step-by-step solution, including the relevant formula and calculations.
  • 00:04:40 ✓ · Gemini Overlay for Messages, YouTube, and PDFs — Dave Burke
    • Dave used the Gemini overlay to: 1) generate a meme about ‘tennis with pickles’ and drag it into a message, 2) ask a question about a YouTube video (‘what is the two bounce rule’), and 3) ask a question about an 84-page PDF rulebook (‘are spin serves allowed’), receiving summaries for each.
  • 00:09:33 ✓ · TalkBack with Gemini Nano — Dave Burke
    • A demonstration of how Gemini Nano provides a much more detailed audio description for an unlabeled image of the Sydney Opera House and a dress being sold online, compared to a basic description.
  • 00:10:25 ✓ · Live Scam Call Detection — Dave Burke
    • A simulated phone call from a ‘bank’ asking to transfer money triggered a real-time ‘Likely scam’ alert on the screen, powered by on-device Gemini Nano.

金句 (6)

  • 00:00:30 — Sameer Samat:

    We’re going even further to make Android the best place to experience Google AI.

  • 00:00:38 — Sameer Samat:

    This new era of AI is a profound opportunity to make smartphones truly smart.

  • 00:00:57 — Sameer Samat:

    So we’ve embarked on a multi-year journey to reimagine Android with AI at the core.

  • 00:03:04 — Sameer Samat:

    I love how it shows how to solve the problem, not just the answer.

  • 00:08:22 — Dave Burke:

    Android is the first mobile operating system to include a built-in, on-device foundation model.

  • 00:11:15 — Dave Burke:

    And everything happens right on my phone, so the audio processing stays completely private to me and on my device.

视觉信号(纯转录看不到的)

屏幕文字时刻 (8)

  • 00:00:04 · Sameer Samat
    • Identifies the first speaker.
  • 00:01:02 · Android AI at the core
    • States the central theme of the presentation segment.
  • 00:01:46 · Circle to Search
    • Brands the feature being discussed.
  • 00:03:29 · Circle to Search Available on more than 100M devices
    • Quantifies the current reach of the feature.
  • 00:03:33 · Circle to Search Available on more than 200M devices by the end of 2024
    • States a forward-looking commitment for feature adoption.
  • 00:04:02 · Introducing Dave Burke
    • Identifies the second speaker.
  • 00:08:46 · Coming to Pixel later this year Gemini Nano with Multimodality
    • Announces a new on-device model and its timeline for Pixel devices.
  • 00:12:20 · Coming tomorrow Android 15 Beta 2
    • Announces the imminent release of the next Android beta.

舞台时刻 (5)

  • 00:00:03 · Sameer Samat walks onto the stage to audience applause.
  • 00:03:36 · Audience applauds after the Circle to Search adoption numbers are shown.
  • 00:03:57 · Sameer Samat introduces and hands off to Dave Burke.
  • 00:04:01 · Dave Burke walks onto the stage to applause.
  • 00:10:52 · The audience laughs and applauds when the on-device scam detection alert successfully interrupts the simulated scam call.

视觉演示 (6)

  • 00:01:56 · Circle to Search on a YouTube video
    • A user’s finger with elaborate nail art circles a pair of bright pink boots on a woman playing guitar in a video. A search panel appears below with shopping results for the boots.
  • 00:02:39 · Circle to Search solving a physics problem
    • A phone displays a digital workbook with a physics problem. The user circles the problem text, and a Google search panel appears with a step-by-step guide to solving it, including identifying the formula and variables.
  • 00:05:05 · Gemini overlay generating an image
    • Within a messaging app, the Gemini overlay is activated. The user types ‘create image of tennis with pickles’. The overlay shows a progress indicator, then displays four generated images. The user drags one of the images directly into the message compose field.
  • 00:05:48 · Gemini overlay summarizing a YouTube video
    • While a YouTube video about pickleball is playing, the Gemini overlay is activated. A chip appears that says ‘Ask this video’. The user taps it and asks a question. Gemini provides a text summary of the answer based on the video’s content.
  • 00:06:54 · Gemini overlay summarizing a PDF
    • While viewing an 84-page PDF rulebook, the Gemini overlay is activated. A chip appears that says ‘Ask this PDF’. The user asks a specific question about a rule, and Gemini provides a concise answer extracted from the document.
  • 00:10:53 · Live scam call detection alert
    • During a simulated phone call, a red alert box pops up over the call screen. It reads ‘Likely scam’ with the text ‘Banks will never ask you to move your money to keep it safe.’ and gives options to ‘Dismiss & continue’ or ‘End call’.

制作信号 (4)

  • 00:00:12 · Switch from live stage shot to a pre-recorded, studio-style shot of the speaker for a cleaner presentation.
  • 00:01:56 · Pre-recorded video montage demonstrating Circle to Search use cases.
  • 00:04:40 · Split-screen view showing the presenter on the left and a high-quality screen recording of the phone UI on the right for the Gemini demo.
  • 00:10:25 · A full-screen graphic simulates an incoming phone call to set up the scam detection demo.

关键主题

Android · Artificial Intelligence · Gemini · On-device AI · Circle to Search · AI Assistant · Context-aware computing · Multimodality · Gemini Nano · Accessibility · TalkBack · Privacy · Scam Detection · Android 15 · Developer Tools

总结要点

  • Google is fundamentally reimagining Android with AI at its core, moving beyond apps to a more integrated, assistive experience.
  • Gemini is becoming the foundational AI assistant for Android, working as a context-aware overlay across the entire OS to provide help without switching apps.
  • On-device AI, powered by Gemini Nano, is a key strategic focus, enabling faster, more private, and offline-capable features like real-time scam detection and enhanced accessibility.
  • Circle to Search is evolving from a simple visual lookup tool into a powerful problem-solving assistant, capable of handling complex homework questions.
  • Android is being positioned as the premier platform for experiencing Google’s most advanced AI, with many features being exclusive to the OS.
  • Multimodality (processing text, image, audio, and speech) is the next frontier for on-device AI, allowing the phone to understand the world more like a human does.
  • Google is emphasizing privacy by running more AI models, like Gemini Nano, directly on the device, so sensitive data like call audio never leaves the phone.