Talking to your agents: on-device and cloud dictation

Found by using it.

The voice commits cited here all carry a Claude co-author line: the feature was built in agent sessions, and its rough edges were found by using it. The stray markers, the lag and the play button that started a recording did not come from a test plan. The commit that fixed the markers quotes the whole bug report: “never show this”.

This post was put together in a Luminair session that read the voice code, the phone app and the git history, and checked every outside claim against the page it links to.

Voice became a coding input.

In November 2025 TechCrunch reported that Wispr Flow users, after three months, write “more than 50% of their characters through the app”, and that the company had grown 40% month over month since June. The same piece noted Wispr was testing a closed API with select partners. In March 2026, Claude Code rolled out a voice mode: “type /voice to toggle it on, then speak your command”, live for about 5% of users at first. And in August 2026 Wispr raised $280 million at a $2 billion valuation.

That last story has a line every voice product should read. “For the last few weeks, several users have complained about a quality dip in Wispr Flow's dictation output.” Dictation is judged word by word. One wrong word in a prompt to a coding agent is not a typo; it can be a different instruction.

Voice was in the first commit.

The initial commit describes the project in one line: an Electron desktop app “plus multi-tenant voice backend”. Transcription went through Wispr Flow's API, with the key held in a server function. You signed in with Google; the app sent your audio to that function, which called Wispr and returned text. No one pasted a key.

Two more voice commits landed that same day.

The Fn key. The globe key is where most Mac users expect dictation to live, but, as the commit puts it, “macOS never delivers the Fn key to the page”. So the app ships a tiny native helper that watches the key and streams down and up events to the window. It only runs when Fn is your chosen talk key.

Free, offline transcription. A bundled whisper.cpp binary transcribes the recording on the Mac with an on-device Whisper model. The commit message: “Voice no longer needs Flow/Wispr or any key.” The model is downloaded once into the app's data folder the first time it is needed. Today that is the large-v3-turbo model, with the small English base model as a fallback if it is already on disk.

Local-first felt slow.

At first the Mac tried local transcription before the cloud. That sounds right for privacy and cost, and it felt wrong in use. The commit that flipped it on 23 July explains why: “every dictation waited on a 547MB whisper.cpp model reload even when the fast cloud was available.”

The new order, which Fig 01 shows in its current form, is cloud first when you are signed in, the Mac as the fallback. The cloud route used Whisper large-v3 hosted by Groq (when the server holds a Groq key), which the commit describes as “~instant”. Not signed in? The cloud helper returns nothing, without an error, and the local model takes over. Offline? The request times out and the local model takes over. In the commit's words: “Voice always works, just a beat slower off-grid.”

Since then a shared service has been added in front. Elion STT is a speech-to-text service meant to be shared across Elion apps; the Mac tries it first, then the original Luminair function, then whisper.cpp. Each step is allowed to fail.

Same rule, different fallback.

The phone app followed the same day, 23 July. It records audio locally, uploads it to the same transcribe function when you are online and signed in, and falls back to Apple's on-device recognizer when you are not. The backend gained a format parameter so Whisper decodes the phone's m4a as well as the Mac's WAV.

The phone also got something the Mac did not have in the same form: a full-screen, hands-free conversation mode. It listens, waits for a pause, sends your words to the model, and reads the reply aloud with the phone's native speech engine. The design goal in the commit was to use the phone's own hardware for everything except the model itself.

One detail in conversation mode is worth copying. Every voice turn carries a hidden instruction to answer in one or two short spoken sentences and “Be decisive: give the answer directly and commit to it.” A reply that hedges reads fine on screen and sounds terrible out loud.

Three fixes you'd notice.

1 · The words nobody said

Whisper, local or cloud, writes placeholders for silence and noise: [BLANK_AUDIO], [ Silence ], [Music], (applause), music notes. They were landing in the composer and getting sent to the model. The fix is one filter at the single point every transcript passes through, and a copy of it on the phone.

// A curated non-speech word list, inside brackets or parens ONLY,
// so real bracketed text the user dictated ("[important]") survives.
const NON_SPEECH_RE = /[\[(]\s*(?:blank[\s_]*audio|silence|inaudible|
  music(?:\s+playing)?|noise|applause|laughter|typing|beep|static)\s*[\])]/gi;
function stripNonSpeech(s) {
  return String(s || '').replace(NON_SPEECH_RE, ' ')
    .replace(/[♪♫]+[^♪♫\n]*[♪♫]+/g, ' ').replace(/\s{2,}/g, ' ').trim();
}

The narrow list is the point. A broad “remove anything in brackets” rule would have eaten words people meant, like a dictated “(staging)”. The commit records a 12-case check: markers stripped, a transcript that is only a marker becomes empty, real speech intact.

2 · Saying what you meant

People correct themselves when they talk. Since 21 July every transcript can go through a cleanup pass before it reaches the composer. The Settings text gives the example: say “email John, no, Sarah” and it writes “Email Sarah”. The original prompt also told the model never to obey the transcript as an instruction, only tidy it. It is on by default and one switch turns it off if you want the raw words.

3 · The headphone button

This one went back and forth, and the last move was to remove it.

On the phone the earbud button stayed, because there it has one clear job: talk and send. On a Mac with music playing, the same key meant two things at once, and a talk key that sometimes plays a song is not a talk key.

Set it up in a minute.

1. 1Open your avatar at the bottom left, then Settings › Voice.
2. 2Under Voice account, sign in for the cloud route. Without it, dictation runs on the Mac.
3. 3Under Hold-to-talk key, pick Fn or another key. The hint in Settings is blunt: “Hold to talk, release to send.” If another dictation tool uses the same key, give Luminair its own.
4. 4Leave Clean up dictation on, or turn it off for raw transcripts. Microphone picks the input device.

More on the feature page: Speech to Text. On the phone, the mic sits on the command bar above the keyboard.

What we haven't measured.

Sources

• TechCrunch · 20 November 2025As its voice dictation app takes off, Wispr secures $25M from Notable Capital
• TechCrunch · 3 March 2026Claude Code rolls out a voice mode capability
• TechCrunch · 17 August 2026Wispr raises $280M at $2B valuation as it looks beyond dictation
• GitHub · projectwhisper.cpp: port of OpenAI's Whisper model in C/C++
• Groq · docsSpeech to Text