I built an AI supercomputer with 5 Mac Studios

About this video

NetworkChuck clusters five Mac Studios (M2 Ultra, 64 GB unified memory each, bought for his studio’s video-editing switch to Mac) using the beta Exo Labs software, aiming to run Meta’s Llama 3.1 405B model locally. He explains parameters as learned knowledge, why bigger models need more VRAM, and how quantization (FP32 down to INT4) shrinks models at a precision cost. The Macs’ unified memory gives the cluster 320 GB of GPU-usable RAM, far cheaper per gigabyte than the 42 RTX 4090s he calculates the 405B would otherwise need, though Nvidia/CUDA still beats Apple/MLX on raw performance. Testing shows networking is the bottleneck: a single Mac ran a small model at about 117 tokens/second (his measurement), but five nodes on 10-gigabit Ethernet dropped to about 29 tokens/second; Thunderbolt bridging helped some. The 70B Llama 3.3 model ran across five nodes at roughly 15 tokens/second, and the quantized 405B did run — at under 1 token/second. He also shows Ollama running 70B and DeepSeek models well on one Mac, and Exo’s OpenAI-compatible API plugging into the Fabric tool.

Key takeaways

• Exo Labs auto-discovers machines on a network and pools their resources into one AI cluster, splitting models across nodes, with a web GUI and an OpenAI-compatible API.
• Apple’s unified memory is the cluster’s real advantage: five 64 GB Mac Studios give 320 GB of GPU-usable memory, versus the creator’s calculation that a 405B model would need 42 RTX 4090s (24 GB VRAM each) unquantized.
• Quantization trades precision for size — the creator cites roughly 0-2% loss at FP16, 1-3% at INT8, and 10-30% at INT4 — which is what makes large models fit consumer hardware at all.
• Interconnect speed dominates cluster performance: adding nodes over 10-gigabit Ethernet made a small model slower (about 117 down to 29 tokens/second in his test), and Thunderbolt (up to 40 Gbps, more direct PCIe access) only partly fixed it.
• The 405B goal was technically met but impractical: the roughly 200 GB quantized model loaded across all five Macs with no swap, yet generated at under 1 token/second, while a single Mac with Ollama handled 70B models comfortably.

Episode details: published February 17, 2025 · 34:57 long. Summary written by Self Growth Videos from the episode itself. Business results and any income figures mentioned are NetworkChuck’s own claims, shared for education — they are not typical or guaranteed.

← Previous: Deep Research…..but Open Source · Next: ChatGPT Operator is expensive….use this instead (FREE + Open Source) →

All NetworkChuck videos · NetworkChuck profile · NetworkChuck on YouTube