Video Understanding

Video understanding is AI that interprets what happens in a video: who appears, what they do and say, what is shown on screen, and when each thing happens. It is used to answer questions about footage, search it, summarize it and, increasingly, to drive editing decisions. There are two broad ways to build it. A multimodal model can "watch" the video directly, ingesting sampled frames and audio. Or a pipeline can extract evidence from the video, such as transcripts, shot boundaries, faces, on-screen text and labeled frames, and a model can reason over that evidence. Knowing when to use which is one of the practical skills of building with video in 2026.

How Models Watch Video

No production model processes every frame. Video is sampled. Google's Gemini API, a widely used native video interface, extracts frames at one per second by default and adds the audio track. Google's documentation puts the cost at roughly 100 tokens per second of video at the default (low) media resolution: 66 tokens per frame plus 32 per second of audio. A one-million-token context therefore holds about three hours of video at low resolution, or one hour at high. Developers can set a custom frame rate and clip to a time range, trading cost against temporal detail.

Other approaches split the job differently:

• Video embeddings for search. TwelveLabs separates retrieval from description. Its models page lists Marengo 3.5, which turns video, audio, images and text into a shared 512-dimensional embedding for search, and Pegasus 1.5, which generates text from video.
• Open vision-language models. Alibaba's Qwen3-VL family (Apache-2.0) advertises a native 256K-token context, expandable to one million, for "hours-long video," with explicit text–timestamp alignment so answers can point to when something happened.
• Specialized detectors. Classic computer vision models for shot boundaries, faces, objects, text recognition and saliency are cheap, fast and often run on the device itself.

Reading Extracted Evidence Instead

The alternative is to turn video into evidence a language model can read and then reason over the evidence:

• Shot and scene boundaries from tools such as PySceneDetect (v0.7.2, October 2026), FFmpeg's scene-change score or the TransNet V2 neural detector (see scene detection).
• Contact sheets: grids of frames chosen for coverage and change, each labeled with its timestamp and often with who is speaking and what they say.
• Transcripts with word timings and speaker turns, from speech recognition and diarization.
• An on-screen index: recognized text, detected faces and people, image labels, and salient regions, which also drive auto reframe.

Evidence is compact. An hour of footage becomes a transcript and a few dozen labeled frames, thousands of tokens instead of hundreds of thousands. It is deterministic, so the same footage gives the same index. It is auditable, because every claim points to a frame or a line. And it can be produced entirely on the device, which keeps private footage private (see edge AI).

When to Watch and When to Read

• Watch when the question is about visual events that no extractor anticipates, such as "when does the dog knock over the vase?". Watch, too, when the content is unknown and needs open-ended description, or when the meaning is in motion rather than speech: sports, physical demonstrations, robotics data.
• Read when the decisions are keyed to speech and structure, as in interviews, talks, podcasts and explainers. Read when timing must be precise to the frame, when the same footage will be queried many times, or when cost and privacy matter.
• Combine them in most real systems. Read evidence first, then pull specific frames or send short clips to a watching model to resolve what the evidence cannot.

The arithmetic explains the pattern. At Gemini's default of about 100 tokens per second, one hour of video is roughly 360,000 input tokens on every query that includes it. A transcript and contact sheet for the same hour can be a small fraction of that and can be reused across every question. See AI model pricing for how token costs compound.

Video Understanding in Editing

Editing is where the read-first approach has proved most useful, because edit decisions must be explainable and frame-accurate. MetaRush, an AI video editing harness, has the agent read extracted contact sheets instead of raw video. Claude Code does the editorial reasoning from frames the command line extracts. Contact sheets pick frames by priority under a cap, starting with one per shot, then visual changes, new speakers and coverage gaps, and each tile is labeled with its time, the current speaker and their words. On a Mac an on-device index built with Apple's Vision framework adds on-screen text, labels, saliency and poses. The skills must pull frames at a visual event before claiming anything about it. That last rule addresses the central failure of agentic video editing: asserting something visual that the model never looked at.

Failure Modes

• Missed events. Anything shorter than the sampling interval, such as a quick cut, a gesture or a flash of text, can fall between frames.
• Visual hallucination. Models describe plausible but absent details, or confuse the order of events. This is the visual form of AI hallucination.
• Text and identity errors. Recognized text and faces are often wrong in motion blur or low light. Face recognition also raises biometric consent questions, covered in data privacy, so names should come from people rather than guesses.
• Context limits. Even million-token contexts force trade-offs between resolution, frame rate and duration.

Where It's Headed

Video understanding is spreading from media into the physical world. The same models that summarize a meeting now label robot training data and feed vision-language-action models and world models. For media, the next steps are streaming understanding that keeps up with live video, better temporal grounding so answers come with exact timestamps, and indexes that make every frame searchable. That last step also matters for discovery, as covered in video GEO.