AI Networking is CRAZY!! (but is it fast enough?)

About this video

This sponsored explainer argues that the network, not the GPUs, is increasingly the bottleneck in AI training. Training clusters of very expensive GPU servers (he cites about $350,000 per server) move hundreds of terabytes of data from storage and constantly exchange data between GPUs, so any straggler creates tail latency; he cites a claim that 33% of AI/ML elapsed time last year was spent waiting on the network. Data centers were built for CPU workloads, and GPU servers also strain power (one server can use a whole rack’s power allotment) and cooling, pushing liquid cooling. InfiniBand was purpose-built for high-performance computing: it avoids TCP/IP overhead, aims for a lossless fabric at 400 gigabit speeds, and uses RDMA to transfer data directly memory-to-memory, bypassing CPUs and OS network stacks. His objections are practical: InfiniBand needs its own switches, NICs and stack, few network engineers know it, compatibility and complexity are painful, and it usually costs more. Ethernet’s answer is RoCE (RDMA over Converged Ethernet), 400 gigabit today, 800 gigabit announced by sponsor Juniper, and 1.6 terabits on the horizon, plus Explicit Congestion Notification and Priority Flow Control working together to approximate a lossless fabric despite Ethernet being inherently lossy. He declares himself Team Ethernet, sketching his own AI data center as a Clos fabric with Juniper 800-gigabit spine/leaf switches and software-defined management to tune those features automatically.

Key takeaways

• AI training is extremely sensitive to latency and packet loss because GPUs work in lockstep, so stragglers create tail latency that slows the whole job
• InfiniBand’s key advantage is RDMA: direct memory-to-memory transfers that bypass the CPU and OS network stack on both servers
• InfiniBand’s downsides are a separate hardware stack, scarce engineer expertise, compatibility complexity, and typically higher cost than Ethernet
• Ethernet counters with RoCE for RDMA, plus ECN marking and PFC pausing to create a near-lossless fabric for AI traffic
• Ethernet speeds are keeping pace: 400 gigabit is current, 800 gigabit was announced by Juniper, and 1.6 terabits per second is on the horizon

Episode details: published February 9, 2024 · 13:41 long. Summary written by Self Growth Videos from the episode itself. Business results and any income figures mentioned are NetworkChuck’s own claims, shared for education — they are not typical or guaranteed.

← Previous: 3 Levels of WiFi Hacking · Next: I ditched my Raspberry Pi for this →

All NetworkChuck videos · NetworkChuck profile · NetworkChuck on YouTube