A voice assistant that answers to your house instead of a data centre is a weekend project, not a research project. With a small computer, a decent microphone and a few open-source pieces, you can set a timer, dim the desk lamp and ask about the weather without a single word leaving your network. Here is how to build it in two days, in the order that saves the most frustration.

What You Gain, and What You Give Up

When the entire chain — wake word, speech recognition, intent matching, spoken reply — runs on hardware in your office, the audio never has to leave the room. That has three practical consequences. It works when your internet does not. It keeps working if a vendor changes its terms or shuts a service down. And there is no monthly fee. The trade is capability: a local stack handles commands and questions about your own devices far better than it handles open-ended trivia. Expect to spend more time on microphones than on software, and expect the first version to be a little deaf. That part is normal.

Choose the Hardware Before You Choose the Software

Three jobs need filling: something to think, something to listen, and something to speak.

The brain

A mini PC with a recent low-power processor and 16 GB of memory is the sweet spot. A Raspberry Pi 4 or 5 with 8 GB handles command-and-control setups perfectly well, and a used office desktop does the same job for less if you can live with the size and the fan noise. If you want the assistant to cope with open questions, add memory and, ideally, a small graphics card; running a language model alongside the speech models is the heaviest part of the load.

The ears

A USB microphone or a small microphone array beats anything built into a laptop. A speakerphone puck designed for conference calls is a good choice on a desk. Place it away from fans, vents and spinning drives — a constant hum is the single biggest cause of misheard commands.

The voice

Any powered speaker will do. Keep it a hand's width from the microphone and turn the volume down; feedback wrecks recognition faster than background noise does.

  • A mini PC, small-form-factor desktop or single-board computer with at least 8 GB of memory
  • A USB microphone or conference speakerphone puck
  • An SSD or a good memory card, plus a network cable if the machine sits near your router
  • A powered USB hub if you plan to add more devices
  • Optional: an ESP32-based satellite for a second room

Saturday Morning: Install the Pieces, Test Them One at a Time

The mistake almost everyone makes is wiring everything together and then guessing which part is broken. Build the chain in pieces.

  1. Install a server operating system — a minimal Linux distribution or a home-automation image — and apply updates before anything else.
  2. Install the automation hub if you want a dashboard for devices and routines. Most voice add-ons plug into it.
  3. Add the four moving parts: a wake-word detector, a speech-to-text engine, an intent handler and a text-to-speech engine. Open-source tools such as openWakeWord, Whisper, Rhasspy and Piper cover all four.
  4. Record a short wav file and push it through the recogniser from the command line. If the transcript is wrong there, no amount of clicking around a settings page will fix it.
  5. Speak a test sentence through the text-to-speech engine. Confirm the audio device names match the ones the software expects.

Saturday Afternoon: Teach the Microphone Some Manners

Signal quality decides everything. Aim the microphone at your face, not at the ceiling. Set the input gain so normal speech peaks well below clipping — if your recording looks like a solid block, it is too hot. Enable noise suppression where your tools offer it, and echo cancellation if the speaker sits close to the mic. Then say your wake word twenty times: from the chair, from the doorway, and while the fan is running. Write down what fails. Change one thing — gain, position or wake-word sensitivity — and test again.

Sunday Morning: Connect Three Things You Actually Use

Resist the urge to automate the whole house. Pick three commands and make them reliable: a timer, a light and a shopping list. Most intent systems work from a small grammar file, so write sentences exactly the way you speak them and point each one at a script. "Add milk to the list" appends to a local to-do file. "Start a twenty-five minute timer" runs a script and announces the end. Keep the grammar tight; a handful of crisp phrases beats two hundred that half-work. Speech recognition also struggles with proper nouns, so add your own device names and street names to the recogniser's vocabulary early.

Sunday Afternoon: Make It Survive Monday

A prototype running in a terminal window dies the first time you reboot. Configure the service to start on boot and restart on crash. Turn on logging and make it rotate, so a failing microphone does not fill the disk. Keep your configuration files in a folder you back up — a version-control repository is ideal — and take a snapshot before you update anything. If privacy was the point of the project, prove it: give the device a fixed address, block its outbound traffic at the router, and confirm your commands still work. Decide one more thing now, especially if anyone else shares the office: tell them a microphone is always listening, and agree on what gets stored.

Where People Get Stuck

  • The wake word fires constantly. Lower the microphone gain, move the device away from the speaker, and raise the detection threshold.
  • Nothing triggers at all. Check that the microphone records at 16 kHz mono; sample-rate mismatches silence more setups than anything else.
  • Replies arrive slowly. Shorten the spoken answer, use a smaller recognition model, and move the machine onto a cable instead of Wi-Fi.
  • Audio cuts out mid-sentence. Suspect power rather than software — a powered hub or a better supply usually cures it.

Living With It for a Week

Keep a note of every command it mishears and add those phrases to the grammar each evening. Treat the setup as something that improves by five minutes a day rather than something you finish. After a month, ask an honest question: is it faster than reaching for your phone? If the answer is yes for three or four everyday tasks, the weekend paid for itself. And keep anything safety-critical — door locks, alarms, anything involving heat or water — on a system you can still override by hand. A voice assistant is a convenience, not a guardian.

Photo: ricardorv30 / Pixabay