AI Dubbing and Lip-Sync

AI dubbing is the automatic translation of a video's speech into another language, voiced by a synthetic copy of the original speaker, often with lip-sync that re-renders the speaker's mouth to match the new words. A process that once needed translators, voice actors and a recording studio can now run in minutes, which has made multilingual publishing practical for individual creators as well as studios. It sits at the meeting point of speech AI, generative video and localization, and it raises the same consent questions as any technology that reproduces a real person's voice and face.

The Pipeline

Most AI dubbing systems chain five steps:

  1. Transcribe the original speech with speech recognition, with word timing and speaker diarization so each voice is handled separately.
  2. Translate the transcript, ideally with awareness of timing: a line that takes two seconds in English may take three in German, so good systems shorten or rephrase to fit.
  3. Synthesize the translation with text-to-speech in a cloned version of the speaker's voice, preserving tone and emotion where possible.
  4. Remix the new voice with the original music and effects, which requires separating the voice from the background first.
  5. Lip-sync, optionally: a video model regenerates the mouth region, or the whole face, so that lip movements match the new language.

Each step can fail independently. Mistranscribed names become mistranslated names; a literal translation runs long and gets sped up; a cloned voice loses the speaker's energy; lip-sync smears teeth or drifts on profile shots. The better products let a person review and edit the transcript and translation before voices are generated.

The Vendors

As of October 2026 the market splits into full-pipeline services, voice platforms and lip-sync specialists:

  • HeyGen offers video translation in more than 175 languages and dialects, with voice cloning, lip-sync and a glossary for brand terms that shouldn't be translated.
  • ElevenLabs Dubbing covers more than 90 languages, with a "cloning strength" control that sets how closely the dub keeps the original voice. It doesn't re-time lips, and ElevenLabs lists its Dubbing Studio editor as being in maintenance mode.
  • Black Forest Labs' FLUX 3 video model, generally available since August 2026, generates lip-synced dialogue in more than 14 languages, an example of lip-sync arriving as a capability of general video models rather than a separate product.
  • Sync (sync.so) sells lip-sync models through an API, used inside other companies' dubbing workflows.
  • Flawless sells "visual dubbing" for film through its TrueSync product; the company says Watch the Skies was the first visually dubbed film released theatrically in the US, and its framework requires the actors' consent.

Platforms Doing It Natively

The largest platforms now dub automatically. YouTube expanded auto-dubbing in September 2024; its help page says dubbing is on by default for eligible creators, translating English videos into 20 languages, and lip-sync, announced at Made on YouTube in September 2025, is described as experimental and limited to select channels. YouTube said that in December 2025 more than 6 million daily viewers watched at least ten minutes of auto-dubbed content. Meta began offering AI translations of Facebook and Instagram Reels in August 2025, starting with English and Spanish and optional lip-sync, and added Hindi and Portuguese in October 2025.

For creators this changes the economics. Dubbing becomes a setting rather than a project, and the question shifts from whether to translate to whether anyone reviews the translation before it reaches an audience in a language the creator doesn't speak.

Dubbing someone in their own cloned voice, and changing their lips to say words they never said, is a form of synthetic media, and the line between localization and a deepfake is consent. Several jurisdictions have written that line into law. Tennessee's ELVIS Act, signed in March 2024, added a person's voice, real or simulated, to the state's right-of-publicity protection. California's AB 2602 and AB 1836, signed in September 2024, made contract terms that allow a performer's digital replica enforceable only with a specific description of the intended uses, and required an estate's consent for replicas of deceased performers. SAG-AFTRA's interactive media agreement, ratified in July 2025, requires consent and disclosure for digital replicas. At the US federal level, the NO FAKES Act was reintroduced in April 2025 and had not been enacted as of October 2026.

In practice, responsible dubbing means three things: permission from the person whose voice and face are reproduced, covering which languages and uses; a record of that permission; and disclosure to viewers that the speech is AI-translated, which platforms increasingly add automatically. See AI content labeling for the platform and EU rules.

Where It's Headed

The direction of travel is toward dubbing as a default publishing option rather than a separate project: a video uploaded once and offered in many languages, with the creator reviewing translations the way they review captions. The open questions are quality in less-resourced languages, whether lip-sync becomes good enough on difficult shots to be the norm, and how voice actors' and performers' rights are protected as synthetic voices take over more dubbing work.

Further Reading