Auto Reframe

Auto reframe is the automatic conversion of video from one aspect ratio to another, most often from 16:9 landscape to 9:16 vertical or 1:1 square, by tracking what matters in each shot and moving a crop window to keep it in frame. It exists because the same footage now has to serve YouTube, TikTok, Instagram Reels and YouTube Shorts, and a static center crop of a landscape shot routinely cuts people out. Auto reframe is now built into most major editors and creator apps, and it is a standard step in turning long recordings into short-form video.

How It Works

Reframing has three stages:

  1. Find the subject. The system detects faces, people and other salient regions in each frame. Face detection and tracking do most of the work for talking-head footage; saliency models, which estimate where a viewer's eye is drawn, handle shots without a clear face.
  2. Plan a path. A crop window of the target aspect ratio is moved to follow the subject. The raw detections jitter from frame to frame, so the path is smoothed, held still when the subject moves only a little, and allowed to move only as fast as a camera operator would.
  3. Respect the cuts. At a shot change the crop should jump to the new subject instead of panning across the edit, so reframing depends on scene detection to know where shots begin and end.

A good reframe feels like a camera operator was following the action. A bad one feels like the frame is chasing the subject, lagging behind it, or drifting while nothing is happening.

The Tools

Reframing became a standard editor feature in 2019 and 2020. Adobe shipped Auto Reframe in Premiere Pro (now called Premiere) in November 2019: it uses Adobe's Sensei models to find the subject, writes ordinary motion keyframes that an editor can adjust, and can be applied to a single clip or to a whole sequence, with slower or faster motion presets. Google open-sourced AutoFlip, a reframing framework built on its MediaPipe pipeline, in February 2020. Apple added Smart Conform to Final Cut Pro 10.4.9 in August 2020; it looks for faces and other areas of interest when a project is reformatted to square or vertical, and the editor fine-tunes the result with the position controls. Blackmagic Design added Smart Reframe, driven by its DaVinci Neural Engine, in DaVinci Resolve 17, in the paid Studio edition.

Creator tools made it a one-click step. CapCut offers auto reframe on desktop and on the web, now as a Pro feature, and clipping services such as OpusClip market reframing that tracks subjects and switches layouts automatically as part of turning a long video into shorts. In these tools reframing rarely appears on its own: it is bundled with highlight selection, captions and layout templates.

Where It Fails

  • Two-shots. Two people sitting apart in a wide shot can't both fit in a vertical frame. The reframe either cuts one person off or swings back and forth between them.
  • Cut-off faces. A fast movement, a person leaning out of frame, or a subject at the edge of the original shot leaves the face partly outside the crop.
  • Text and graphics. On-screen text, slides and lower thirds in the original are cropped away, since trackers follow faces, not words.
  • Restless framing. Following every small movement produces a nauseating, constantly moving frame. Smoothing and a "hold still" zone are what make a reframe watchable.
  • Wrong subject. In a crowd or a busy scene, saliency picks something bright or moving that isn't the point of the shot.

Split Layouts

When one crop can't hold the story, the alternative is to show more than one crop at once. Vertical video of conversations commonly stacks two crops, one per speaker, or puts the speaker in one half and screen content or B-roll in the other. Podcast clips use the same approach with each participant's camera. Choosing between a single tracked crop, a stacked layout and a letterboxed full frame is an editorial decision, and the better tools make it per shot rather than once for the whole video. Layouts also have to leave room for the platform's interface; captions and graphics placed where an app draws its buttons are hidden. See auto-captions for safe areas.

Reframing and AI Editing

For AI video editing systems, reframing is where computer vision and editorial judgment meet. Knowing who is speaking, from speaker diarization, lets the crop follow the speaker rather than whoever moves most. Storing the crop path as editable data, rather than baking it into pixels, lets a person correct one shot without redoing the rest. And checking the result, for example flagging any moment where the main face is less than fully inside the frame, catches the failures above before a video ships. As with other video understanding tasks, the hard cases are not technical but editorial: what the shot is about.

Further Reading