Scene Detection (Shot Boundary Detection)

Scene detection, more precisely called shot boundary detection, is the task of finding the points in a video where one shot ends and the next begins. It's one of the oldest problems in computer vision for video, and still one of the most useful: it turns a continuous file into a list of shots that can be indexed, searched, summarized or re-edited. Editors use it to cut a flattened master back into its original shots, for example to rebuild an edit decision list from a finished video; search and recommendation systems use it to pick thumbnails and index moments for video search; and AI systems that read video use it to decide which frames are worth looking at.

What Counts as a Boundary

The difficulty is that "the picture changed a lot" has several causes, and only some are shot boundaries:

• Hard cuts: an instant change from one shot to another between two frames. These are the easiest to find.
• Gradual transitions: dissolves, fades to and from black, wipes. The change is spread over many frames, so no single pair of frames differs much.
• Camera and subject motion: fast pans, zooms, whip moves, someone walking past the lens. These can change the frame as much as a cut does, without being one.
• Lighting events: camera flashes, explosions, a light switched on. A sudden brightness change that isn't an edit.

A good detector finds the first two and rejects the last two, and in practice most errors come from confusing motion and flashes with cuts, or from missing slow dissolves.

Classic Methods

The traditional approach compares consecutive frames with a cheap measure and flags a boundary where the difference spikes. Common measures are the average change in pixel values, the difference between color histograms, and the change in edges. Thresholds can be fixed, or adaptive, comparing each frame's change to the recent average so that a busy, handheld shot doesn't trigger constant false cuts. Fades are often handled separately by watching for frames that approach a flat black or white level.

The open-source PySceneDetect library is a widely used implementation of these methods. As of October 2026 its current release is 0.7.2 (12 October 2026), and it ships five detectors: ContentDetector (frame-to-frame change in color and brightness), AdaptiveDetector (the same signal compared with a rolling average, to suppress false cuts during fast motion), ThresholdDetector (fades to and from black), HistogramDetector (color histograms) and HashDetector (perceptual hashes of each frame). FFmpeg exposes a per-frame scene-change score, from 0 to 1, through the scene variable of its select filter, and a dedicated scdet filter that reports a score for each frame; both are common in scripts that pick thumbnails or split footage before processing.

Editors build it in as well. DaVinci Resolve Studio lists scene cut detection among its Neural Engine tools, used to split a flattened master back into shots. Final Cut Pro 12.3 (June 2026) added Edit Detection, which finds the hard cuts in an already-edited video and splits it into clips; Apple notes that it may miss fades.

Learned Detectors

Learned detectors, trained with deep learning on labeled transitions, handle gradual transitions and motion better than thresholds do. TransNet V2 (Souček and Lokoč, 2020) is a compact convolutional network that looks at a window of frames and predicts, for each frame, the probability that it is a shot boundary; it remains the reference model in the field, although its repository has had no updates since late 2023. AutoShot (CVPR 2023) introduced a dataset of short-form videos and a model found by neural architecture search that outperformed TransNet V2 on it, a reminder that vertical, fast-cut short-form video behaves differently from film and television.

A practical design combines the two: a fast classic detector for hard cuts, plus denser sampling at low resolution to catch gradual changes and bursts of motion, with a learned model reserved for difficult material.

Scene Detection in AI Video Pipelines

Shot boundaries are a building block for video understanding. A multimodal model that is asked about a long video can't look at every frame affordably, so a pipeline can sample frames per shot rather than at fixed intervals, building contact sheets that show each shot once. In AI video editing, the same boundaries tell an editing system where the footage already cuts, so that an auto-reframe path or a trim doesn't straddle two shots, and so that automated QC can tell whether black or a sudden change in a render was in the source footage all along. It's often worth keeping three kinds of signal separate rather than collapsing them into "scenes": hard cuts, gradual changes, and bursts of camera or subject movement, since each means something different to an editor.

Limits

• Shots aren't scenes. A scene in the storytelling sense (one place, one time, one dramatic unit) usually contains many shots. Grouping shots into scenes needs semantic understanding, not just pixel change.
• Thresholds are content-dependent. Music videos, sports and screen recordings each need different settings.
• Re-encoded and letterboxed video introduces compression artifacts and borders that detectors can mistake for content changes.