Passthrough

Passthrough (also called video see-through or video passthrough) is the technique of displaying a real-time camera feed of the physical world inside a VR headset, enabling mixed reality experiences without optical transparency. Rather than looking through a transparent lens (as in AR glasses), users see the real world through cameras mounted on the headset's exterior, with digital content composited into the video feed.

Passthrough has become the dominant approach to mixed reality in current-generation headsets. Apple Vision Pro, Meta Quest 3, and Meta Quest Pro all use high-resolution passthrough as their primary mixed reality mechanism. The approach has significant advantages over optical see-through: digital content can fully occlude real-world objects (enabling opaque virtual objects), the entire visual field is under software control (allowing sophisticated blending and effects), and HDR processing can enhance the real-world view.

The technical requirements are demanding. Latency is the critical constraint: any delay between head movement and the video feed update causes motion sickness and disorientation. Modern passthrough systems target under 10ms motion-to-photon latency, requiring high-speed cameras (typically 60-90 Hz or higher), fast image processing pipelines, and predictive reprojection algorithms that warp the camera image based on estimated head motion.

Resolution and color fidelity must approach natural vision quality to avoid the sensation of looking through cameras. Apple Vision Pro's dual 6.5µm micro-OLED displays and high-resolution camera array produce passthrough that approaches (though doesn't quite match) natural sight. Meta Quest 3's passthrough is lower fidelity but sufficient for spatial awareness and short mixed reality sessions.

Depth estimation enables realistic digital object placement. Stereo cameras provide disparity-based depth, while LiDAR sensors (used in Vision Pro) provide direct depth measurements. Accurate depth maps allow digital objects to be correctly occluded by real-world geometry — a virtual ball rolling behind a real table disappears appropriately.

The passthrough approach versus optical see-through (like waveguide-based AR glasses) represents a fundamental architectural choice in spatial computing. Passthrough offers superior digital content quality and full environmental control but requires wearing an opaque headset. Optical see-through preserves natural vision and enables all-day wearable form factors but constrains digital content brightness, opacity, and field of view.

The trajectory suggests both approaches will coexist: passthrough headsets for immersive work, entertainment, and high-fidelity mixed reality; optical see-through glasses for lightweight, always-on ambient computing. As passthrough fidelity improves toward perceptual transparency and AR glasses gain wider FOV and brighter displays, the experiences they enable will converge.