How Claude Code orchestrated the Gemini stack to turn a discharge note into a film.

Plan the storyboard

Discharge note in. Structured 10-scene JSON out. The system instruction forces a 2:1 patient-demonstration-to-doctor ratio and bans invented dosages.

"""You are a medical patient-education film director. Given a hospital
discharge note, produce a short film (8-12 scenes, each one continuous
8-second shot) that VISUALLY DEMONSTRATES every actionable instruction.

CORE PRINCIPLE: This is a demonstration film, not a lecture. Aim for
2 PATIENT demonstration scenes for every 1 DOCTOR scene.

Structure:
- Scene 1 (doctor): warm intro naming the diagnosis.
- Middle scenes: every concrete instruction gets its OWN dedicated
  patient demonstration scene. Break multi-step actions into multiple
  scenes (crutch walking vs. crutch stairs are separate).
- Final scene (doctor): gently remind the patient about red flags.
  Keep this scene calm and reassuring, NOT dramatic.

Visual prompt rules (these become Veo prompts — make them VIVID):
- One continuous shot, no cuts, no on-screen text.
- Patient scenes: describe the EXACT physical motion — body position,
  hand placement, the object being manipulated, the camera angle.
- 8 seconds is short — pick ONE clear motion per scene.

Never invent dosages, drug names, or actions not in the discharge note.
Return strict JSON matching the schema."""

{
  "title": string,
  "scenes": [{
    "character": "doctor" | "patient",
    "visual_prompt": string,
    "narration":    string,
    "caption":      string
  }]
}

Render each scene as 8-second video

Each scene's visual_prompt + the style suffix is sent to Veo with the matching character image as the first frame. The image anchors face / style consistency across all 10 scenes.

# scripts/veo_client.py
op = client.models.generate_videos(
    model="veo-3.0-generate-001",
    prompt=scene.visual_prompt + STYLE_SUFFIX,    # style chosen per run
    image=types.Image.from_file(reference_image), # doctor.jpg OR patient.jpg
    config=types.GenerateVideosConfig(
        aspect_ratio="16:9",
        duration_seconds=8,
        number_of_videos=1,
        person_generation="allow_adult",
    ),
)
# poll the long-running op every 15s up to 600s

realistic: "photorealistic cinematic shot, soft natural lighting,
            warm and calm tone, single continuous 8-second take"

animation: "Pixar-style 3D animated film, smooth cel-shaded
            rendering, expressive friendly character animation,
            vibrant but gentle color palette"

anime:     "hand-drawn 2D Japanese anime illustration, soft
            watercolor backgrounds, Studio Ghibli short film tone"

Synthesize narration

The narration field from each scene gets routed to one of two voices based on who is speaking. Each ≈20-word line lands in roughly 8 seconds at speaking_rate=0.95, matching the clip duration.

# scripts/tts_client.py
DOCTOR_VOICE  = "en-US-Chirp3-HD-Charon"   # warm male
PATIENT_VOICE = "en-US-Chirp3-HD-Aoede"    # calm female

response = client.synthesize_speech(
    input=tts.SynthesisInput(text=scene.narration),
    voice=tts.VoiceSelectionParams(
        language_code="en-US",
        name=DOCTOR_VOICE if scene.character == "doctor" else PATIENT_VOICE,
    ),
    audio_config=tts.AudioConfig(
        audio_encoding=tts.AudioEncoding.MP3,
        speaking_rate=0.95,
    ),
)

Composite + caption + concatenate

Pillow renders each caption as a transparent PNG strip (Homebrew's ffmpeg ships without freetype, so drawtext isn't available). ffmpeg overlays the strip, muxes the MP3, then concatenates all 10 scenes into the final MP4.

# scripts/ffmpeg_stitch.py
caption_png = _render_caption_png(scene.caption)  # Pillow → 1280×96 PNG

ffmpeg -i raw_video.mp4 -i narration.mp3 -i caption.png \
  -filter_complex "[0:v]scale=1280:720,pad=...[v0];
                   [v0][2:v]overlay=0:574[outv]" \
  -map [outv] -map 1:a \
  -c:v libx264 -c:a aac -shortest scene_NN.mp4

ffmpeg -f concat -i scenes.txt -c:v libx264 -c:a aac final.mp4