Speech Enhancement

Speech enhancement is the use of signal processing and machine learning to make recorded speech clearer: removing background noise, echo and room reverberation, separating a voice from music or other voices, and in its newest form regenerating the voice as if it had been recorded in a studio. It's a staple of creator tools, because most video and podcast audio is recorded in kitchens, cars and conference rooms rather than booths. It's also one of the places where AI quietly changes the content of a recording, which makes it worth understanding what each kind of enhancement actually does.

Three Kinds of Enhancement

Noise and echo suppression keeps the original recording and removes what isn't speech. Classic methods estimated a noise profile and subtracted it; modern ones use neural networks that predict, for each moment and frequency band, how much of the signal is voice, and attenuate the rest. These systems run in real time on ordinary hardware and are built into video-call software. Because they only remove energy, they can make a voice sound thin or watery when pushed hard, but they don't add sounds that weren't there.

Voice isolation and source separation splits a recording into stems, such as voice, music and effects, so the voice can be kept and the rest lowered or discarded. It's the tool for a speaker recorded over music or at a loud event.

Generative restoration goes further. Instead of cleaning the recorded waveform, a generative model listens to the degraded speech and produces a new, clean rendition: full bandwidth, no room sound, studio presence. The results can be striking on poor recordings, because the model can restore frequencies a phone microphone never captured. But the output is a new performance synthesized from the old one.

The Tools

Open-source models cover the first and third kinds, though both of the best-known projects have been quiet for some time. DeepFilterNet, a real-time noise-suppression model published at ICASSP 2022 with later versions in 2022 and 2023, is open source; its latest release, 0.5.6, dates from August 2023. Resemble Enhance, released by Resemble AI in December 2023, has two stages: a denoiser that cleans the recording, and a generative enhancer, based on conditional flow matching, that restores distortion and extends bandwidth. Both still work, but neither has had a stable release since 2023.

Commercial tools span all three kinds:

  • Adobe Podcast Enhance Speech is the best-known "make it sound like a studio" tool. Version 2 (November 2024) added a strength control, and a March 2026 update added source separation, with downloadable speech, noise and music stems.
  • ElevenLabs Voice Isolator (July 2024, with an API) separates speech from background noise, music and ambient sound.
  • DaVinci Resolve Studio has included an AI Voice Isolation track effect since version 18.1 (November 2022).
  • NVIDIA Broadcast added Studio Voice, which makes a webcam microphone sound like a studio microphone, as a beta in version 2.0 (January 2025), running locally on RTX graphics cards.
  • Descript Studio Sound applies AI voice enhancement inside Descript's editor.

The Risk of Changed Words

A generative enhancer is, in effect, a speech synthesizer conditioned on a noisy recording. Where the input is clear, it reproduces what was said. Where the input is ambiguous, buried in noise, clipped or mumbled, it has to fill the gap, and it fills it with something plausible. That can mean a slightly different vowel, a changed consonant, a dropped word or an invented syllable, delivered in the speaker's own voice with full confidence. Researchers now study this directly. A 2025 paper accepted at AAAI 2026 describes two kinds of hallucination in generative speech enhancement: linguistic errors, where the spoken content changes, and acoustic errors, where the speaker's voice characteristics change, both worse at low signal-to-noise ratios. Users have reported the same thing in practice, including enhanced dialogue that contained words nobody said.

For entertainment content this may be acceptable. For interviews, journalism, legal or medical recordings, and anything where a person's exact words matter, it is a real problem, and one the speaker may never notice. The same concern applies to speech recognition, where models are known to produce fluent text that was never said, a form of AI hallucination.

Using Enhancement Safely

  • Keep the original. Treat enhanced audio as a new asset alongside the source, never a replacement, so the edit can switch back. This is non-destructive editing applied to audio.
  • Use the least invasive tool that works. Suppression first; generative restoration only when the recording is otherwise unusable.
  • Verify the words. Transcribe the original and the enhanced version and compare them. Wherever the words differ, listen, and prefer the less processed take for that stretch.
  • Disclose when it matters. If a voice has been substantially regenerated, that is synthetic media, and some contexts call for saying so; see AI content labeling.
  • Mind the level. Enhancement changes loudness; re-measure before delivery (see loudness normalization).

Where It Fits

Speech enhancement sits between capture and editing in modern AI video editing pipelines, usually as an optional step per speaker. It increasingly overlaps with text-to-speech and voice cloning, since the most powerful restorers and the best synthetic voices use the same generative models. That overlap is the reason the line between "cleaned up" and "re-voiced" deserves an explicit check rather than an assumption.

Further Reading