Cloning my Voice Into an AI Assistant

NetworkChuck Technology Careers Career & Money

About this video

Chuck sets out to give Terry, his completely local AI voice assistant from a previous video, a cloned voice using free, local, open-source tools instead of a paid cloud service like ElevenLabs. The workflow centers on Piper: collect clean voice samples, format them as a folder of short wav files plus a metadata.csv pairing each file with its transcription, fine-tune an existing Piper model rather than training from scratch, then export the result to the ONNX format so it can be used in Home Assistant or other text-to-speech programs. He shows two ways to get data: recording prompted phrases in Piper Recording Studio (his editor Mike recorded about 200 phrases, and Chuck later recorded about 400 for a better version of his own voice), or downloading his own YouTube audio with yt-dlp, removing music in Audacity, stripping silence with FFmpeg, cutting clips to no longer than 15 seconds, and transcribing them locally with Whisper. Training is demonstrated on a laptop with an Nvidia 3080 in WSL, on Terry’s dual-4090 server, and on a cloud GPU instance he estimates at about five dollars an hour, with warnings about pinned older library versions and a CUDA issue affecting 4090 cards. Results vary sharply with data quality: Mike’s voice sounds convincing, early attempts at Chuck’s fast-talking YouTube voice sound poor, and cleaner studio recordings work much better. He finishes by loading the model into Home Assistant via a Samba share and the Piper/Wyoming integration and testing the new assistant live.

Key takeaways

Piper fine-tuning starts from an existing checkpoint instead of training from scratch, and the finished voice is exported as an ONNX model file plus its JSON config
Training data must be clean, mono, short clips (no longer than 15 seconds) paired with accurate transcriptions in a metadata.csv file
Piper Recording Studio gives an easy path for your own voice; more and cleaner utterances (about 400 in his redo) noticeably improve quality
YouTube-sourced audio needs music and silence removed and automatic Whisper transcription before it is usable as a dataset
Voice quality depends mainly on clip cleanliness, quantity, and varied speech, not just on GPU power or number of training epochs

Episode details: published November 19, 2024 · 37:11 long. Summary written by Self Growth Videos from the episode itself. Business results and any income figures mentioned are NetworkChuck’s own claims, shared for education — they are not typical or guaranteed.

← Previous: my local, AI Voice Assistant (I replaced Alexa!!) · Next: I started a blog…..in 2024 (why you should too) →

All NetworkChuck videos · NetworkChuck profile · NetworkChuck on YouTube