Audio-to-Video Generation

Audio-to-video generation is the creation of video from a soundtrack: a voice recording, a song or a script read aloud, where the audio's timing, rhythm and emotion decide what appears on screen. It reverses the usual order of AI video. In text-to-video, a model generates pictures from a prompt and the sound is added afterwards or generated alongside; in audio-to-video, the sound comes first and is fixed, and the pictures are made to fit it. The category runs from the narrow (a talking head whose lips and expressions follow a voice) to the broad (a music video or short film whose shots, characters and pacing are inferred from a song), and it is one of the newer frontiers of AI filmmaking.

Audio-Driven Characters

The narrow version, animating a person from audio, has the longest research history. Alibaba's EMO (February 2024) generated expressive portrait video directly from audio with a diffusion model, without intermediate 3D faces or landmarks. Fudan University's Hallo series followed through 2024, extending to hour-long, 4K output and, with Hallo3, moving to a video diffusion transformer. ByteDance's OmniHuman-1 (February 2025) generated human video from audio, pose or both in a single diffusion transformer, and OmniHuman-1.5 (August 2025) added guidance from a multimodal language model so that gestures and expressions follow the meaning of what is said. MultiTalk (May 2025) handled several people in one shot, binding each audio stream to the right person, and InfiniteTalk (August 2025) streamed audio-driven motion for long videos.

Open weights arrived with Alibaba's Wan2.2-S2V (August 2025), a 14-billion-parameter speech-to-video model released under Apache-2.0 that animates characters to speech or singing, with the clip's length following the input audio. On the commercial side, avatar platforms such as Synthesia and HeyGen turn a script, or in some cases a recording, into a presenter video; their current avatar models, HeyGen's Avatar V (April 2026) and Synthesia's Express-3 (July 2026), are the polished, business-facing form of this narrow version. See digital humans and Synthesia vs HeyGen.

Audio-Driven Storytelling: Koyal

The broad version treats the audio as a script for a whole film. Koyal (koyal.ai) is the clearest example as of October 2026. It describes itself as an "agentic AI filmmaking platform": it takes a script, a voiceover or a song, reads emotion, pacing and tone from the audio, and produces a film or music video with consistent characters and settings across shots. It also offers multilingual script-to-audio-to-video and lip-sync. Koyal was founded by siblings Mehul Agarwal (CEO) and Gauri Agarwal (CTO), Carnegie Mellon alumni, and is listed by Y Combinator in its Fall 2025 batch. It debuted publicly at the WAVES 2025 summit in Mumbai in May 2025 with an album of music videos made with well-known Indian musicians, and opened a public beta in September 2025. The company says it has run paid pilots with music labels and film studios.

Koyal is also notable for building consent into the product. Before a person's likeness can be used to create a character, its CHARCHA check ("Computer Human Assessment for Recreating Characters with Human Actions") asks them to perform randomized live actions on a webcam, such as turning their head, squinting or smiling, so that someone can't upload another person's photos. The method was described in a paper presented in the NeurIPS 2024 Creative AI track. It is a practical model for likeness consent of the kind discussed in AI content labeling.

How a Pipeline Works

Turning a song or narration into a film is less a single model than a planning problem, which is why these tools describe themselves as agentic. A pipeline has to:

1. Analyze the audio: transcribe any words with timings, find the tempo, beats and sections of music (verse, chorus, bridge), and estimate the emotional arc.
2. Plan the shots: decide how many shots there are and how long each lasts, aligning cuts with beats and phrases, and assign each shot a subject, setting and camera move.
3. Fix the characters: build reference images or a fine-tuned model for each character, so the same person looks the same in every shot. This is where consent checks belong.
4. Generate the shots with a video model, using an audio-driven model for shots where someone sings or speaks on camera.
5. Assemble the edit against the fixed soundtrack, where the music sets the timing and the pictures follow, the reverse of ordinary editing, in which music is fitted to picture.

This differs from the native audio that many 2025 and 2026 text-to-video models now produce. Those models generate sound along with the picture; audio-to-video takes an existing recording, often one with rights and an artist attached, and treats it as the thing that must not change.

Uses and Limits

The clearest demand is from musicians and labels, for music videos, lyric videos and visualizers that would otherwise be too expensive to make for every track, and from creators who want narrated stories, podcasts or audiobooks turned into watchable video. It overlaps with generative music, since a generated song can now come with a generated video, and with AI dubbing, which uses the same lip-sync models in reverse.

The limits are the ones that run through all AI video. Keeping characters consistent across dozens of shots is still hard. Lip-sync on singing, with held notes and wide mouth shapes, is harder than on speech. Rights don't transfer: generating pictures for a song doesn't license the song, and a recognizable face needs its owner's permission. And because the output is synthetic media of real-looking people, it falls under the labeling rules that platforms and the EU now enforce.