Run your own AI (but private)

About this video

NetworkChuck demonstrates running a private, local AI on his own computer with no data sent to a company, and argues this matters for people whose employers block ChatGPT for privacy and security reasons. He explains what an LLM is using Hugging Face and Meta’s Llama 2 as the example, noting the creator’s figures that Llama 2 was trained on over 2 trillion tokens and over a million human-annotated examples, and the creator’s estimate that training used over 6,000 GPUs, 1.7 million GPU hours, and around $20 million. He installs Ollama — directly on macOS, and on Windows via WSL with Ubuntu — then runs Llama 2 7B and Mistral locally, noting GPUs and Apple M-series chips are much faster than CPUs alone. Because the base models hallucinate details about him, he explains two ways to teach a model your own data: fine-tuning, illustrated with VMware’s effort to teach a model about an unreleased vSphere version using, in the creator’s example, 9,800 examples that changed 65 million parameters, and RAG (retrieval-augmented generation), where the model consults a vector database before answering. He closes with a bonus PrivateGPT setup using WSL, NVIDIA drivers, and Poetry to chat with his own markdown journals.

Key takeaways

• Ollama is the installer/runner used in the video: install it, then run a model such as Llama 2 with a single ollama run command; Windows users are directed through WSL first
• Local models still hallucinate — Llama 2 and Mistral both gave wrong names for NetworkChuck — so private data needs fine-tuning or RAG, not just a local download
• Fine-tuning is far cheaper than pre-training, per the creator’s VMware example: a small set of prompt/answer examples changed under 1% of a 7-billion-parameter model in a few minutes
• RAG does not retrain the model; it connects the model to a database of documents and instructs it to check that database before answering
• VMware Private AI with NVIDIA, and alternatives with Intel and IBM/Red Hat mentioned in the video, are pitched as packaged on-prem stacks bundling deep-learning VMs and tools like PyTorch and TensorFlow for companies

Episode details: published March 12, 2024 · 22:13 long. Summary written by Self Growth Videos from the episode itself. Business results and any income figures mentioned are NetworkChuck’s own claims, shared for education — they are not typical or guaranteed.

← Previous: I ditched my Raspberry Pi for this · Next: 60 Hacking Commands You NEED to Know →

All NetworkChuck videos · NetworkChuck profile · NetworkChuck on YouTube