Why LLMs get dumb (Context Windows Explained)

About this video

Chuck explains why LLMs seem to ‘get dumb’ in long conversations: context windows. An LLM’s context window is its short-term memory, measured in tokens — how the model counts text, not words or characters — and everything counts toward it: your messages, its replies, hidden system prompts, pasted documents and code. In LM Studio he loads Gemma 3 4B with a 2,048-token window, tells it he is reading ‘How to Take Smart Notes’, fills the window with a story and sequels past 100%, and the model can no longer recall the book; reloading with 4,096 tokens, it remembers and apologises. Bigger is not free for local models: officially about 128k tokens for that model, but loading the full window on his RTX 4090 with 24GB VRAM maxed the GPU and made responses crawl, because attention work grows with context. For cloud models he cites, as figures stated in the video, GPT-4o at 128k, Claude 3.7 at 200k, Gemini 2.5 at 1 million, and an added note that Meta’s Llama 4 Scout claims 10 million. Even huge windows do not guarantee recall: he summarises the ‘Lost in the Middle’ paper’s U-shaped result — models attend best to the start and end of context and drop off in the middle — and gives a simplified account of self-attention assigning relevance scores to words on every turn, which is why long, topic-hopping chats get slower and more hallucination-prone. His practical rule is to start a new chat on a significant topic shift. Optimisations he demos are Flash Attention (processing tokens in chunks without storing the full comparison matrix), K/V cache compression/quantisation — enabling both let him load the full 128k window without maxing VRAM and with much better speed — and paged cache, which spills attention cache to system RAM at a speed cost. He also warns larger windows enlarge the attack surface for hidden malicious prompting, and includes a sponsored tip to convert webpages to clean markdown with the Jina reader (r.jina.ai prefix) before pasting them in.

Key takeaways

• A context window is token-limited short-term memory; system prompts, documents and code fill it as well as chat messages, and overflow pushes out early facts.
• Chuck demonstrates forgetting live: at 2,048 tokens Gemma 3 loses the book title he gave it; at 4,096 tokens it retains it.
• Large local context costs VRAM and compute — loading ~128k tokens maxed his 24GB RTX 4090 and slowed generation sharply.
• ‘Lost in the Middle’: accuracy is best for information at the beginning and end of context and worst in the middle, even inside the advertised window.
• Mitigations covered: start a new chat when the topic shifts, and for local models use Flash Attention plus K/V cache quantisation (paged cache trades speed for capacity).

Episode details: published April 9, 2025 · 15:18 long. Summary written by Self Growth Videos from the episode itself. Business results and any income figures mentioned are NetworkChuck’s own claims, shared for education — they are not typical or guaranteed.

← Previous: everyone is putting AI in schools…… · Next: Saying Goodbye…. →

All NetworkChuck videos · NetworkChuck profile · NetworkChuck on YouTube