Speech Recognition (ASR)

Speech recognition, or automatic speech recognition (ASR), is the conversion of spoken audio into text, and increasingly into text with structure: punctuation, word-level timestamps and labels for who is speaking. It is the foundation layer of captions, transcript-based video editing, meeting notes, search over audio and voice agents, and the step where spoken language enters natural language processing. A broader overview of the field is at speech and voice AI. This page focuses on how recognition itself works in 2026, how it is measured, and where it still fails.

How Modern ASR Works

Three families of models dominate as of October 2026:

  • Encoder–decoder transformers. OpenAI's Whisper, released in September 2022 and trained on 680,000 hours of weakly supervised audio, is the reference design. An encoder reads a log-mel spectrogram of up to 30 seconds of audio, and a decoder writes text tokens one at a time, like a language model conditioned on sound. The open weights have not changed since large-v3 (November 2023) and large-v3-turbo (October 2024), which cut the decoder from 32 layers to 4 for speed. Whisper covers about 99 languages under the MIT license.
  • Transducers. NVIDIA's Parakeet TDT models pair an acoustic encoder with a small prediction network and emit tokens in step with the audio frames. "TDT" stands for token-and-duration transducer: the model predicts each token and how long it lasts, which gives precise word timing. Parakeet TDT 0.6B v2 (May 2025) is English-only, and v3 (August 2025) covers 25 European languages. Both are released under CC-BY-4.0.
  • LLM-based recognizers. A speech encoder feeds a large language model that writes the transcript. NVIDIA's Canary-Qwen-2.5B joins a FastConformer encoder to Qwen3-1.7B. These models handle context and vocabulary well, but they inherit the language model's tendency to produce fluent text the audio does not support.

How It's Measured

The headline metric is word error rate (WER): substitutions plus deletions plus insertions, divided by the number of words in a reference transcript. WER depends heavily on text normalization (casing, punctuation, numbers written as digits or words), so numbers from different evaluations rarely compare cleanly. Two other measures matter for real products. The real-time factor is processing time divided by audio duration. Timestamp accuracy is how close each word's start and end are to where it was actually spoken, and it decides whether captions and cuts land correctly.

Hugging Face's Open ASR Leaderboard is the most-cited public comparison. Its accompanying paper covers 86 open and proprietary systems on 12 datasets. The board was rebuilt in 2026 around cleaned versions of its test sets, so older WER figures quoted for Whisper or Parakeet are not comparable with today's. As of October 2026, proprietary systems lead with average WER below 4%, the best open models, NVIDIA's Canary-Qwen and Parakeet among them, sit around 5 to 5.5%, and Whisper large-v3 trails them.

Failure Modes

  • Hallucination. Autoregressive decoders can write text that was never spoken, especially over silence, music or noise. The 2024 FAccT paper "Careless Whisper" found hallucinated phrases in roughly 1% of Whisper transcriptions it studied, and 38% of those hallucinations included explicit harms. This is the speech version of AI hallucination.
  • Loops and drops. Long audio can trigger repeated phrases. Some models also silently skip stretches of speech.
  • Names and jargon. Proper nouns remain the most common residual errors even when overall WER is low.
  • Real-world audio. Overlapping speakers, room echo, phone audio, accents and switching between languages all raise error rates. Most benchmarks under-represent them.

Word Timing and Hallucination: Evidence From a Bake-off

For editing, where every cut and caption depends on word boundaries, the differences between model families show up clearly. In one video harness's local transcription bake-off, MetaRush's tests of October 2026 on an Apple M2 Max, the measured results were:

  • Parakeet TDT 0.6B v2 (int8, through sherpa-onnx on the CPU, in chunks of up to 30 seconds cut at silences) scored 0.8% WER on a 199-second public-domain speech recording and 0.6% and 0.9% on 7.9- and 31.5-minute mixes. It emitted no words at all over digital silence, ambient noise or a music bed, and ran at a real-time factor of about 0.03 without using the GPU.
  • whisper.cpp 1.9.5 with large-v3-turbo scored 1.1–1.3% on the long mixes but wrote "Thank you." over pure silence and over the music bed. It also placed a word 1.02 seconds early, inside a deliberate 1.5-second pause. Without the option that stops it conditioning on previous text, it fell into a repetition loop at the end of the 7.9-minute mix.
  • Parakeet TDT 0.6B v3 dropped whole sentences, repeatably and depending on how the audio was chunked: 47 of 373 words in a single pass over the speech recording, and 63 of 965 words on the 7.9-minute mix in 30-second chunks. The same mix in 20-second chunks lost 4.

The test's own caveats apply. Its synthetic fixtures were clean text-to-speech, its real speech was studio-quality single speakers, and real two-person interviews were not tested. The result was English Parakeet v2 as the default, with Whisper as the multilingual fallback behind a speech-detection gate. More detail is in Whisper vs Parakeet.

Open Models Versus APIs

Open models run locally, which suits private footage and edge AI deployment. whisper.cpp (v1.9.5, October 2026) runs Whisper efficiently on laptops, and sherpa-onnx (v1.13.8, September 2026, Apache-2.0) runs Parakeet, speaker diarization and voice activity detection through ONNX Runtime. See open-weight models for the broader trade-offs.

APIs lead on accuracy and convenience, and they change quickly:

  • OpenAI launched gpt-transcribe on July 28, 2026 and has since deprecated gpt-4o-transcribe, which shuts down February 26, 2027. Its documentation states that "the timestamp_granularities[] parameter is only supported for whisper-1", so word timestamps depend on whisper-1, which shuts down the same day. Developers building captions or editing tools on OpenAI need a plan for that date.
  • AssemblyAI's Universal-3.5 Pro (July 2026) handles code-switching across 18 languages, labels speakers, and costs $0.21 an hour for batch transcription.
  • Deepgram's Nova-3 serves batch and streaming transcription. Flux, which it calls "the first conversational speech recognition model built specifically for voice agents," detects the end of a speaker's turn.
  • ElevenLabs' Scribe v2 covers more than 90 languages, with word timestamps and diarization of up to 32 speakers.

Two of these are compared head to head in Deepgram vs AssemblyAI.

Choosing a Recognizer

Start with the job. Captions and transcript-based editing need accurate word timing and no invented words over silence. Voice agents need low latency and turn detection. Archives need language coverage and cost efficiency. Then guard against known failures. Gate output with a speech detector, clamp timestamps to the audio's length, check for missing stretches of speech, and assign speakers with a separate diarization pass when the recognizer does not do it. Whatever the benchmark says, test on a sample of your own audio before committing.

Further Reading