Whisper (OpenAI)

Whisper is OpenAI's open-weight speech recognition model family, released in September 2022 and still the most widely used open model for multilingual transcription. One model handles transcription in 99 languages, translation of speech into English and language identification, and its permissive license and huge ecosystem of ports, such as whisper.cpp, have made it the default starting point for developers. As of October 2026 its open weights are unchanged since large-v3 (2023) and large-v3-turbo (2024), and OpenAI's own transcription API has moved on to newer models.

How It Works

Whisper is an encoder-decoder transformer. The encoder reads a 30-second window of audio as a log-Mel spectrogram, and the decoder writes the transcript token by token, like a language model conditioned on sound. OpenAI's paper, "Robust Speech Recognition via Large-Scale Weak Supervision", trained it on 680,000 hours of multilingual audio collected from the web, which is why it copes with accents, background noise and many languages without fine-tuning. Sizes run from tiny to large; large-v3 has 1.55 billion parameters. Large-v3-turbo keeps the large-v3 encoder but cuts the decoder from 32 layers to 4, giving 809 million parameters and much faster decoding, at a small cost in quality and without translation.

Strengths and Weak Spots

• Breadth. 99 languages and direct translation into English, more than any rival open model family.
• Ecosystem. The reference Python package, whisper.cpp (version 1.9.5, October 6, 2026) for CPUs and Apple and other GPUs, WhisperX for forced alignment, and ports for almost every platform, which makes it practical for edge AI.
• License. MIT for code and weights per OpenAI's repository (Hugging Face lists large-v3 as Apache-2.0); both permit commercial use without attribution beyond the notice.
• Word timing. Whisper does not predict per-word timestamps natively. The reference implementation derives them by aligning cross-attention with dynamic time warping, whisper.cpp labels its option experimental, and WhisperX adds a separate alignment model.
• Hallucinations. A decoder asked to write something sometimes writes fluent text that was never said, especially over silence or music. The "Careless Whisper" study (FAccT 2024) found hallucinated text in 1.4% of 13,140 audio segments, and 38% of those hallucinations contained harmful content such as invented violence. The usual mitigations are voice activity detection before transcription and not conditioning each window on the previous text (see AI hallucinations).

Whisper in OpenAI's API

OpenAI's hosted whisper-1 model is being retired: it is scheduled for removal on February 26, 2027, and it is the only OpenAI transcription model that returns word-level timestamps. OpenAI's recommended model is now gpt-transcribe, released on July 28, 2026, which replaced gpt-4o-transcribe. Teams that rely on word timing from OpenAI's API need to plan a migration, either to a self-hosted Whisper with alignment or to another provider.

How It Compares

On the Hugging Face Open ASR Leaderboard (version dated October 17, 2026), Whisper large-v3 averages 7.16% word error rate on English and turbo 7.64%, behind NVIDIA's Parakeet TDT 0.6B v2 at 6.02%, which is also roughly eight to thirteen times faster by the leaderboard's throughput measure and returns native word timestamps. Proprietary systems lead the board. Whisper keeps its advantage in language coverage, and on several European languages large-v3 beats Parakeet's multilingual v3. Hosted APIs such as Deepgram, AssemblyAI and ElevenLabs' Scribe add streaming, diarization and support. A common pattern in 2026 is a faster English model with Whisper as the multilingual fallback. See Whisper vs Parakeet, open-weight models and speech and voice AI.