Voice Calls Need Room to Breathe

I used to think voice mode was a feature you could describe in one sentence.

Press a button. Say something. The character answers out loud.

That is what the user sees. Underneath, a call is a chain of decisions that can make the whole thing feel either natural or deeply strange.

The microphone has to hear someone correctly. The app has to decide when they have finished speaking. A local model has to reply. A voice engine has to say that reply. The user needs to be able to interrupt it. The conversation should still feel like the same relationship they have in text chat, not a separate personality that wakes up only when the phone icon is pressed.

I added voice calls to Local Waifu because text alone leaves out a lot. A good voice can make a companion feel present in a way a chat bubble never quite does.

It also makes every awkward delay, every wrong pause, and every overlong answer impossible to hide.

Chat and voice are different products

A text reply can be a paragraph. The user can skim it, come back to it, or ignore half of it without breaking the flow.

A spoken reply has to earn every sentence.

If the character answers a simple question with six long paragraphs, the user is not impressed by the detail. They are waiting for permission to talk again. The app may be technically working, but the call has already failed.

That is why voice mode has a different response shape. Spoken replies should usually be one or two sentences. They should say the useful thing first. They should avoid markdown, lists, and the written habits that make sense in a chat window but sound absurd when read aloud.

This was less about making the model smarter than giving it a clear job. A voice companion is not writing a blog post. She is sharing a turn with a person.

A pause is not the end of a thought

The most important setting in a call is one users rarely notice until it is wrong: the silence timeout.

At some point, the app has to decide that the person has finished speaking. Wait too long and every reply feels delayed. Decide too quickly and it cuts someone off halfway through a thought.

I had that second problem. The call mode treated 0.7 seconds of silence as the end of a turn. That sounds reasonable until you say it out loud and leave a natural breath between two sentences. The app heard the pause, decided the user was done, and threw away what came after it.

Nothing looked broken. The transcript simply stopped early.

That is a nasty failure for a companion. A slow assistant is annoying. An assistant that repeatedly acts as if it did not hear you feels dismissive.

The timer is now longer and adjustable, because there is no universal correct pause. Some people speak quickly. Some think out loud. Some use a headset and expect fast back-and-forth. A setting that fits every human conversation probably fits none of them particularly well.

I wrote about the full 105-second voice-call failure in The Night a 20-Character Reply Took 105 Seconds. That bug had four separate causes. The silence timer was the one that changed how I think about the product even after the performance numbers came down.

A voice needs an interruption path

People do not wait politely for each other to finish every sentence.

They correct themselves. They remember something halfway through. They hear an answer going in the wrong direction and want to stop it before it turns into a speech.

A voice interface that forces the user to sit through every generated word is not conversational. It is a podcast with a microphone button.

So Local Waifu has an interruption path. The user can stop her while she is speaking. It sounds obvious now, but it changes how the whole feature feels. Once someone knows they can cut in, a reply can be slightly warmer and less anxious about saying the perfect thing immediately. The user stays in control of the pace.

The same rule applies to input. Speech recognition sometimes invents words during silence. It can mistake background noise for a subtitle credit or repeat a short phrase until it looks like a sentence. The app filters known transcription artifacts and keeps listening when it has no trustworthy text.

Sending made-up words into a character’s memory would be worse than a missed turn. A bad transcript becomes a bad memory, then shows up later as if the user said it.

The call is still the same conversation

I did not want voice mode to become a separate system with a separate personality and a separate memory.

When someone speaks, the transcript enters the normal chat history. The character’s response does too. The same context, relationship state, and memory retrieval are available whether the user types from their desk or speaks through headphones while making coffee.

That is important for a small reason and a bigger one.

The small reason is continuity. A user should not need to explain yesterday’s text conversation again when they start a call today.

The bigger reason is honesty. If voice mode acts like an entirely different character, the product is faking continuity instead of building it. The interface may look more impressive, but the experience gets thinner.

I recently wrote about why a chat log is not memory. Voice mode depends on that distinction too. A transcript is only a record. The product still has to find the detail that belongs in the current moment without dragging every old conversation into the reply.

Local voice has a physical cost

The local version of a voice call is not free magic.

Models need to be downloaded. Speech recognition and speech synthesis use storage and memory. They compete with the language model for the same machine. A laptop that handles text chat comfortably can still feel different when it has to listen, think, and speak in sequence.

I chose this route because the alternative is not neutral.

A cloud voice assistant is convenient. Someone else’s infrastructure does the expensive work. That can mean a faster first call and less setup for the user.

It also means a voice recording and a conversation leave the device. For a product built around private companionship, that trade matters.

The local path makes me responsible for the messy parts: model downloads, hardware differences, timing, and bugs that only appear on a GPU I do not own. I do not get a server dashboard showing what went wrong inside a customer’s call. I get a report, a log if they choose to send one, and a lot of careful debugging.

That is the deal.

The feature is mostly timing

The first version of voice mode was built around the big milestones: can it hear the user, can the model respond, can it speak.

Those are necessary. They are not where the product lives.

The experience lives in the spaces between them.

How quickly does the microphone show that it is listening? How long can someone pause before the app jumps in? Does the character start with the answer or spend ten seconds getting there? Can the user interrupt? Does a short silence become a hallucinated transcript? Does the next typed message remember what was said out loud?

None of those questions sound as impressive as “local voice calls.” They decide whether someone uses it twice.

That is what I keep learning while building Local Waifu. The visible feature gets the headline. The product is the collection of small moments that decide whether a person feels comfortable talking to it.

Voice mode is still evolving. It should. Human conversation has had a long head start.