What happens between you speaking and text landing in your app: what each speech model actually does, where your own words get in, and how Memory learns from the fixes you make.
Pick an engine, and whether Memory already knows your words. The transcripts are real outputs from the local engines, fed synthetic speech from macOS say.
“Please deploy the dashboard to Supabase and ping Kubernetes on Vercel.”
Every engine turns sound into text, but they're built very differently, and that difference decides whether your words can be “suggested” to it before it listens.
Chops audio into 30-second windows, turns each into a spectrogram, and an encoder summarises it. A decoder, a small language model, then writes the transcript one token at a time, each guess conditioned on the text before it.
whisper-server (whisper.cpp), or cloud APIsWalks through the audio frame by frame and, at each step, emits a token or moves on. Parakeet's TDT also predicts how many frames to skip, which is why it's fast. There is no “previous text” input, so there's nowhere to put a prompt.
sherpa-onnx websocket servers (offline, and online for live typing)Predicts the whole transcript in one pass rather than word by word, with language set to auto and number formatting (“twenty” → 20) built in. Very fast; like the transducers, it has no text context to steer.
sherpa-onnx offline serverThe decoder has just “read” your words, so when the audio sounds like “super base”, the spelling it saw a moment ago is the likely next token. The language token (<|en|>) is guessed before your words are added, so hints don't sway language detection: we measured the same 0.998 “Chinese” confidence with and without a prompt. Hints can still pull the writing toward English, which is what section 07 guards against.
The model library is the set of downloaded model files: Whisper's ggml-*.bin in ~/.cache/openwhispr/whisper-models/, and Parakeet, X-ASR and SenseVoice under parakeet-models/. They never change as you use the app. Everything personal lives in the word library, which has three shelves:
| Shelf | Who fills it | Stored in | What it holds |
|---|---|---|---|
| Memory | Learned from your fixes, or added in the Memory view | whisperwoof-vocabulary.json |
The right spelling and how it was misheard (“alternatives”), how often each fix happened, and which apps you use it in. |
| Dictionary | You (Settings → Dictionary); auto-learn also adds new words | SQLite custom_dictionary |
Plain words. Also shown to the polish model. |
| Word Packs | Curated lists shipped with the app. Off by default since 1.22.0. | resources/vocabulary-packs/ |
Brands & Products, Everyday Essentials, Food & Drink (266 words). |
With encryption on (Settings → Encryption), Memory is stored encrypted as whisperwoof-vocabulary.json.wwenc and the SQLite database is encrypted too. Word Packs ship with the app and hold nothing personal.
Why packs are off now. Generic words in every prompt pull Whisper toward them. With Brands & Products on, “Vercel” came out as “Versacell” (from “Versace”). Turn on the packs you actually talk about.
| Engine | Hints | Swaps | Polish |
|---|---|---|---|
| Whisper, localwhisper-server | Yespinned language, or Auto with the guard in 07 | Yes | Yes |
| Parakeet TDT | Nohotwords measured: no effect | Yes | Yes |
| X-ASR (live typing), SenseVoice | No | Yeson the final text | Yes |
| Cloud, batchOpenAI · Groq · Mistral | Yes | Yes | Yes |
| OpenWhispr Cloud, streaming providers | Partlystreaming gets the Dictionary only | Not yet | Yes |
Whisper reads at most 224 tokens of prompt and keeps the last 224, silently dropping the start. A token is a word-piece; for these word lists it's about 3 characters. Token counts below come from whisper.cpp's own tokenizer run on the real word lists.
macos-text-monitor, needs Accessibility) watches that text field for 30 seconds.Why ask? A swap edits every future transcript, so one wrong rule does damage everywhere. Memory only offers a swap when it's a pattern (the same fix twice) and the misheard text couldn't be something you really said: “Superbase” and “shunade” aren't words, while “their”, “bills”, “10” and “a team” are, so those are never offered, though the fixed word still goes into your Dictionary and hints. And even then, nothing is swapped until you say so. The app checks single words against the Mac's 235,000-word dictionary; this demo uses a small built-in list.
With the dictation language on Auto, an English word list once turned Chinese speech into an English translation: a clip of “现在天气很好” came out as “Now the time is good”. So hints used to be switched off entirely in Auto mode. Now they're on, with a guard that uses the one thing hints can't sway: Whisper's language detection.
| Piece | File |
|---|---|
| Every transcript goes through here: swaps, then polish | src/helpers/audioManager.js · processTranscription |
| Hint prompt: priority, 560-char budget, your words last | src/whisperwoof/core/vocabulary/pack-manager-pure.js · buildSttPrompt |
| Swap rules and applying them | src/whisperwoof/core/vocabulary/memory-replacements.js |
| Finding misheard → corrected pairs | src/utils/correctionLearner.js · extractCorrectionPairs |
| Memory storage (learn, undo, delete) | src/whisperwoof/bridge/vocabulary.js + vocabulary-pure.js |
| Watching the pasted field | src/helpers/textEditMonitor.js + resources/macos-text-monitor.swift |
| Auto-mode guard | src/whisperwoof/core/language/auto-hints.ts |
| Learning glue, undo, delete sync | src/helpers/ipcHandlers.js · _learnCorrections, _getSttHintPrompt |