# Video Understanding > How AI models understand video (frame sampling, token costs, embeddings), and when reading extracted evidence such as contact sheets works better. Source: https://metavert.io/video-understanding Published: 2026-10-20 Updated: 2026-10-20 **Video understanding** is AI that interprets what happens in a video: who appears, what they do and say, what is shown on screen, and when each thing happens. It is used to answer questions about footage, search it, summarize it and, increasingly, to drive editing decisions. There are two broad ways to build it. A [multimodal](https://metavert.io/multimodal-ai) model can "watch" the video directly, ingesting sampled frames and audio. Or a pipeline can extract evidence from the video, such as transcripts, shot boundaries, faces, on-screen text and labeled frames, and a model can reason over that evidence. Knowing when to use which is one of the practical skills of building with video in 2026. ### How Models Watch Video No production model processes every frame. Video is sampled. Google's Gemini API, a widely used native video interface, extracts frames at one per second by default and adds the audio track. Google's documentation puts the cost at roughly 100 tokens per second of video at the default (low) media resolution: 66 tokens per frame plus 32 per second of audio. A one-million-token context therefore holds about three hours of video at low resolution, or one hour at high. Developers can set a custom frame rate and clip to a time range, trading cost against temporal detail. Other approaches split the job differently: - **Video embeddings for search.** TwelveLabs separates retrieval from description. Its models page lists Marengo 3.5, which turns video, audio, images and text into a shared 512-dimensional [embedding](https://metavert.io/embeddings) for search, and Pegasus 1.5, which generates text from video. - **Open vision-language models.** Alibaba's Qwen3-VL family (Apache-2.0) advertises a native 256K-token context, expandable to one million, for "hours-long video," with explicit text–timestamp alignment so answers can point to when something happened. - **Specialized detectors.** Classic [computer vision](https://metavert.io/computer-vision) models for shot boundaries, faces, objects, text recognition and saliency are cheap, fast and often run on the device itself. ### Reading Extracted Evidence Instead The alternative is to turn video into evidence a language model can read and then reason over the evidence: - **Shot and scene boundaries** from tools such as PySceneDetect (v0.7.2, October 2026), FFmpeg's scene-change score or the TransNet V2 neural detector (see [scene detection](https://metavert.io/scene-detection)). - **Contact sheets**: grids of frames chosen for coverage and change, each labeled with its timestamp and often with who is speaking and what they say. - **Transcripts** with word timings and speaker turns, from [speech recognition](https://metavert.io/speech-recognition) and [diarization](https://metavert.io/speaker-diarization). - **An on-screen index**: recognized text, detected faces and people, image labels, and salient regions, which also drive [auto reframe](https://metavert.io/auto-reframe). Evidence is compact. An hour of footage becomes a transcript and a few dozen labeled frames, thousands of tokens instead of hundreds of thousands. It is deterministic, so the same footage gives the same index. It is auditable, because every claim points to a frame or a line. And it can be produced entirely on the device, which keeps private footage private (see [edge AI](https://metavert.io/edge-ai)). ### When to Watch and When to Read - **Watch** when the question is about visual events that no extractor anticipates, such as "when does the dog knock over the vase?". Watch, too, when the content is unknown and needs open-ended description, or when the meaning is in motion rather than speech: sports, physical demonstrations, robotics data. - **Read** when the decisions are keyed to speech and structure, as in interviews, talks, podcasts and explainers. Read when timing must be precise to the frame, when the same footage will be queried many times, or when cost and privacy matter. - **Combine** them in most real systems. Read evidence first, then pull specific frames or send short clips to a watching model to resolve what the evidence cannot. The arithmetic explains the pattern. At Gemini's default of about 100 tokens per second, one hour of video is roughly 360,000 input tokens on every query that includes it. A transcript and contact sheet for the same hour can be a small fraction of that and can be reused across every question. See [AI model pricing](https://metavert.io/ai-model-pricing) for how token costs compound. ### Video Understanding in Editing Editing is where the read-first approach has proved most useful, because edit decisions must be explainable and frame-accurate. MetaRush, an AI video editing harness, [has the agent read extracted contact sheets instead of raw video](https://metavert.io/metarush). Claude Code does the editorial reasoning from frames the command line extracts. Contact sheets pick frames by priority under a cap, starting with one per shot, then visual changes, new speakers and coverage gaps, and each tile is labeled with its time, the current speaker and their words. On a Mac an on-device index built with Apple's Vision framework adds on-screen text, labels, saliency and poses. The skills must pull frames at a visual event before claiming anything about it. That last rule addresses the central failure of [agentic video editing](https://metavert.io/agentic-video-editing): asserting something visual that the model never looked at. ### Failure Modes - **Missed events.** Anything shorter than the sampling interval, such as a quick cut, a gesture or a flash of text, can fall between frames. - **Visual hallucination.** Models describe plausible but absent details, or confuse the order of events. This is the visual form of [AI hallucination](https://metavert.io/ai-hallucinations). - **Text and identity errors.** Recognized text and faces are often wrong in motion blur or low light. Face recognition also raises biometric consent questions, covered in [data privacy](https://metavert.io/data-privacy), so names should come from people rather than guesses. - **Context limits.** Even million-token contexts force trade-offs between resolution, frame rate and duration. ### Where It's Headed Video understanding is spreading from media into the physical world. The same models that summarize a meeting now label robot training data and feed [vision-language-action models](https://metavert.io/vision-language-action-models) and [world models](https://metavert.io/world-models). For media, the next steps are streaming understanding that keeps up with live video, better temporal grounding so answers come with exact timestamps, and indexes that make every frame searchable. That last step also matters for discovery, as covered in [video GEO](https://metavert.io/video-geo). ## Related Topics - [Multimodal AI](https://metavert.io/multimodal-ai) — Models that combine vision, audio and text - [Computer Vision](https://metavert.io/computer-vision) — The underlying perception techniques - [Scene Detection (Shot Boundary Detection)](https://metavert.io/scene-detection) — Finding cuts and changes - [Agentic Video Editing](https://metavert.io/agentic-video-editing) — Where understanding drives edits - [Speech Recognition (ASR)](https://metavert.io/speech-recognition) — The audio half of understanding - [Vision-Language-Action Models](https://metavert.io/vision-language-action-models) — Understanding that drives robots - [Video GEO](https://metavert.io/video-geo) — Making video findable by AI ## Further Reading - [Video understanding (Gemini API)](https://ai.google.dev/gemini-api/docs/video-understanding) — Google; sampling rates, token costs and limits - [TwelveLabs models](https://www.twelvelabs.io/product/models-overview) — Marengo embeddings and Pegasus video-to-text - [Qwen3-VL](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct) — Alibaba's open vision-language models - [PySceneDetect](https://github.com/Breakthrough/PySceneDetect) — open-source shot boundary detection - [TransNet V2](https://github.com/soCzech/TransNetV2) — a neural shot boundary detector