Acquired NVIDIA Part II: The Machine Learning Company (2006-2022)

Category: Acquired Podcast (Deep Dives) · Duration: 135 min · ▶ Watch

Speakers: Ben Gilbert and David Rosenthal

Switch language → 中文

Segments (27)

  • 00:00 · Introduction & The Scale of Compute
    • The hosts introduce the episode and discuss the unfathomable scale of computations required for modern AI.
  • 07:00 · Nvidia’s Early Strategy & Execution
    • A recap of Nvidia’s early days, focusing on their aggressive 6-month product cycles and decision to write their own drivers.
  • 15:00 · The Stanford Researcher & Early GPU Compute
    • The story of a researcher using GPUs for quantum chemistry, highlighting the early potential of GPUs for general compute.
  • 21:00 · The Creation of CUDA
    • Jensen Huang’s massive bet on creating CUDA, a software layer to make GPUs programmable for general tasks.
  • 24:00 · The 2008 Crash & Staying the Course
    • Nvidia’s stock plummets 80% during the financial crisis, but the company remains committed to its expensive CUDA strategy.
  • 36:00 · The Tegra Mobile Experiment
    • Nvidia’s attempt to enter the mobile processor market with Tegra, which ultimately found success in the Nintendo Switch.
  • 43:00 · ImageNet and the AI Big Bang
    • The creation of the ImageNet dataset by Fei-Fei Li, setting the stage for the deep learning revolution.
  • 46:00 · AlexNet Changes Everything
    • Alex Krizhevsky and team use Nvidia GPUs to train a neural network that shatters previous ImageNet records.
  • 45:50 · The AlexNet Breakthrough
    • The hosts discuss how the AlexNet team used deep learning and GPUs to achieve a massive breakthrough in AI image recognition.
  • 48:21 · CUDA and the Search for a Market
    • Nvidia’s development of CUDA and cuDNN provided the essential software layer that made GPUs accessible for AI researchers.
  • 51:41 · The Trillion Dollar AI Market
    • The realization that AI could revolutionize digital advertising and content aggregation created a massive new market for Nvidia’s hardware.
  • 54:41 · Nvidia’s Stock Journey and Crypto Boom/Bust
    • A look back at Nvidia’s stock price history, including the massive run-up and subsequent crash driven by cryptocurrency mining demand.
  • 57:21 · Parallel Computing: Graphics, AI, and Crypto
    • The core realization that graphics, neural networks, and crypto mining all rely on ‘embarrassingly parallel’ matrix math perfectly suited for GPUs.
  • 01:10:21 · The Rise of the Data Center Segment
    • Nvidia’s Data Center revenue exploded, matching and eventually surpassing its core gaming business due to enterprise AI demand.
  • 01:20:21 · The Mellanox Acquisition and Data Center Scale
    • Nvidia acquired Mellanox to control the high-speed networking required to connect thousands of GPUs, making the data center the new unit of compute.
  • 01:24:21 · The Failed Arm Acquisition and Grace CPU
    • Despite failing to acquire Arm, Nvidia pushed forward with developing its own Arm-based data center CPU, Grace.
  • 01:34:21 · Nvidia’s Financials and Future Outlook
    • An analysis of Nvidia’s high valuation, massive free cash flow, and strategy to sell enterprise software solutions.
  • 91:40 · The Omniverse Vision
    • The hosts discuss Nvidia’s ambitious goal to simulate the physical world using their hardware and software.
  • 94:55 · Vanta Sponsorship
    • An ad read for Vanta, a compliance and security platform.
  • 97:40 · Recap of Nvidia’s Early Days
    • A brief review of Nvidia’s near-death experiences and early strategies in the graphics card market.
  • 100:30 · The Birth of CUDA
    • Discussion on Nvidia’s massive bet to make GPUs programmable for general-purpose computing.
  • 106:00 · The Cost of Innovation
    • How building CUDA added significant costs to chips without immediate benefit to their core gaming customers.
  • 111:00 · Market Crash and Skepticism
    • Nvidia faces a massive stock drop amid the 2008 financial crisis and Wall Street’s doubt about CUDA’s ROI.
  • 115:00 · The Mobile Misadventure
    • Nvidia’s failed attempt to dominate the smartphone market with their Tegra chips.
  • 119:15 · Hyperscaler Capital Expenditure
    • Analyzing the massive investments made by tech giants in data center infrastructure.
  • 125:00 · The AI Big Bang
    • How the AlexNet breakthrough proved that GPUs were the perfect hardware for deep learning.
  • 130:35 · Carve Outs
    • The hosts share their personal recommendations for books, cameras, and experiences.

Specific Prices (12)

Timestamp Item Value Context
13:45 Nvidia Market Cap $20 billion Nvidia’s peak market capitalization in mid-2007 before the financial crisis.
23:45 ATI Acquisition $6-7 billion The estimated price AMD paid to acquire ATI, Nvidia’s main graphics competitor.
25:00 Nvidia Stock Drop 80% decline The massive drop in Nvidia’s stock price during the 2008 financial crisis.
54:41 Nvidia Stock (2012-2015) ~$5 The price of Nvidia stock before the market realized its potential in AI.
54:55 Nvidia Stock (Current at recording) ~$220 The stock price at the time the podcast was recorded.
55:18 Nvidia Stock (2018 Peak) $65 The peak stock price during the 2017/2018 cryptocurrency mining boom.
55:33 Nvidia Stock (2019 Trough) $34 The stock price after the crypto crash.
01:06:43 RTX 3090 GPU ~$2,000 The price of a high-end consumer graphics card.
01:07:06 H100 GPU $20,000 - $30,000 The estimated price of a single enterprise data center GPU.
01:10:00 Mellanox Acquisition ~$7 Billion The price Nvidia paid to acquire networking company Mellanox in 2020.
111:30 AMD acquisition of ATI ~$6-7 billion The price AMD paid to acquire Nvidia’s main graphics competitor, ATI.
112:30 Nvidia Market Cap Dropped from ~$20B to ~$4B Nvidia’s valuation plummeted 80% during the 2008 financial crisis and due to market skepticism.

Memory Facts (4)

  • [19:50] Employees dedicated to the CUDA platform
    • 1,100 employees
  • [30:00] Number of cores on a modern consumer GPU
    • Over 10,000 cores
  • [46:41] Deep learning algorithms require massive amounts of compute.
    • Described metaphorically as needing compute on the order of ‘grains of sand on earth’.
  • [94:10] Training a single speech recognition model takes more math operations than grains of sand on Earth.
    • Math operations vs. grains of sand

Bottleneck Claims (3)

  • [16:30] Programming languages were the bottleneck for GPU compute.
    • Evidence: Researchers had to translate complex math into graphical shader languages (CG) just to use the GPU hardware.
  • [46:41] Deep learning was bottlenecked by computational power before GPUs were utilized.
    • Evidence: The algorithms existed for decades but were impractical to run on traditional CPU architectures.
  • [01:06:21] The current bottleneck in computing is moving workloads off the CPU.
    • Evidence: The industry shift towards specialized accelerators (GPUs, DPUs) to handle tasks CPUs are too slow for.

Predictions (4)

  • [22:30, Long-term (played out over 10+ years)] If Nvidia builds a general-purpose compute platform (CUDA), developers will eventually find uses for it.
  • [01:00:21, Future/Ongoing] There will be more massive markets discovered that rely on parallel matrix multiplication.
  • [01:33:21, Long-term] Nvidia is targeting a $1 Trillion total addressable market.
  • [93:30, Long-term future] Nvidia’s Omniverse will allow for the simulation of the entire physical world.

Key Technologies (13)

  • Programmable Shaders: Allows developers to write custom code to dictate how pixels and vertices are rendered, moving away from fixed-function graphics.
  • CUDA (Compute Unified Device Architecture): A parallel computing platform and API model that allows software developers to use a CUDA-enabled GPU for general purpose processing.
  • Tegra: Nvidia’s system-on-a-chip (SoC) series designed for mobile devices, integrating an ARM architecture processor and Nvidia GPU.
  • Convolutional Neural Networks (CNNs): A class of deep neural networks, most commonly applied to analyzing visual imagery, which proved highly effective when run on GPUs.
  • Deep Learning / Neural Networks: A branch of AI that uses multi-layered algorithms to learn from large amounts of data.
  • CUDA: Nvidia’s parallel computing platform and programming model that allows developers to use GPUs for general-purpose processing.
  • cuDNN: A GPU-accelerated library of primitives for deep neural networks, built on top of CUDA.
  • Transformer Models: Advanced AI architectures used for tasks like natural language processing and image generation.
  • NVLink / Infinity Fabric: High-speed interconnect technologies used to link multiple chips together with low latency.
  • DPU (Data Processing Unit): A specialized processor designed to handle data center networking and data movement tasks.
  • CUDA (Compute Unified Device Architecture): A parallel computing platform and programming model that allows developers to use GPUs for general-purpose processing.
  • Tegra: Nvidia’s system-on-a-chip (SoC) series developed primarily for mobile devices.
  • Deep Learning: A subset of machine learning based on artificial neural networks that requires massive parallel compute power.

Companies Mentioned (19)

Intel · Microsoft · AMD · Nintendo · Google · Facebook (Meta) · Baidu · Cerebras · Tesla · Mellanox · Arm · TSMC · Vanta · Nvidia · ATI · Apple · Qualcomm · Microsoft, Amazon, Meta, Alphabet · Softbank

Notable Quotes (7)

Thank you, I can get my life’s work done in my lifetime. — Jensen Huang (paraphrased) @ 15:30

If you don’t build it, they can’t come. — Jensen Huang (paraphrased) @ 22:30

Embarrassingly parallel. — Ben Gilbert @ 30:45

We cannot overstate the importance of this moment… This was the Big Bang moment for artificial intelligence. — Host @ 48:00

If we were a hedge fund, we’d put all our money into Nvidia. — Host (quoting Marc Andreessen) @ 54:08

You say solutions, I hear gross margin. — Host @ 01:21:08

If you don’t build it, they can’t come. — Host (paraphrasing Jensen Huang) @ 108:20

Key Topics

The historical evolution of Nvidia from a gaming company to an AI infrastructure giant. · The strategic importance and massive financial risk of developing the CUDA platform. · The serendipitous intersection of GPU architecture and the computational needs of deep learning. · The impact of the AlexNet breakthrough on the tech industry. · Artificial Intelligence · GPU Computing · Nvidia Corporate History · Data Center Infrastructure · Semiconductor Industry · Nvidia's strategic long-term bets · The development and financial burden of CUDA · Nvidia's failure in the mobile chip market · The intersection of GPUs and the rise of Deep Learning · The massive scale of hyperscaler capital expenditure

Takeaways

  • Nvidia’s current dominance in AI was not an overnight success, but the result of a decade-long, highly risky investment in the CUDA software ecosystem.
  • Hardware alone is insufficient; Nvidia’s moat is built on the software layer (CUDA) that makes their hardware accessible to researchers and developers.
  • The AI revolution was catalyzed when researchers realized that the ‘embarrassingly parallel’ nature of graphics processing perfectly matched the math required for neural networks.
  • Nvidia’s GPUs, originally designed for graphics, proved to be the perfect hardware for the parallel processing required by deep learning.
  • The creation of CUDA and cuDNN created a massive software moat that locked developers into Nvidia’s ecosystem.
  • Nvidia successfully transitioned from a gaming-focused company to a data center and AI powerhouse.
  • The acquisition of Mellanox allowed Nvidia to scale their architecture from individual chips to entire data centers.
  • Nvidia’s current dominance in AI is the result of a decade-long, highly criticized investment in making GPUs programmable via CUDA.
  • Strategic failures, such as Nvidia’s attempt to enter the mobile market, can provide valuable lessons and technology that pivot into future successes.
  • Building a robust software ecosystem around hardware creates a significantly deeper competitive moat than hardware alone.