Diffusion Transformers (DiT)

Diffusion transformers (DiT) are diffusion models whose denoising network is a transformer instead of the convolutional U-Net used by earlier image generators. The image or video is compressed into a latent representation, cut into patches, and treated as a sequence of tokens; the transformer learns to remove noise from those tokens step by step, guided by a text prompt or other conditioning. Since 2024 the design has become the dominant architecture for text-to-video and high-end image generation, because it scales predictably with compute and handles space and time in the same way.

From U-Nets to Transformers

The first widely used latent diffusion models, including Stable Diffusion, denoised with a U-Net, a convolutional network borrowed from image segmentation. In December 2022, William Peebles and Saining Xie showed in "Scalable Diffusion Models with Transformers" that a plain transformer operating on patches of the latent could replace it. Their key finding was about scaling: models with more compute per step, whether from more depth and width or from more (smaller) patches, consistently produced better images, and their largest model, DiT-XL/2, set a then state-of-the-art FID of 2.27 on class-conditional ImageNet at 256×256. The paper also introduced adaLN-Zero, a way of feeding the noise level and class into each block through its normalization layers, which many later models adopted.

Video followed quickly. OpenAI's Sora report in February 2024 described a transformer trained on "spacetime patches" of compressed video, across varied durations, resolutions and aspect ratios. A month later, the Stable Diffusion 3 paper combined a multimodal DiT (MM-DiT), which keeps separate weights for text and image tokens but lets them attend to each other, with rectified-flow training, and showed the approach scaling for text-to-image as well.

How a DiT Works

A diffusion transformer has four parts:

1. A compressor. A variational autoencoder (VAE) maps pixels into a much smaller latent tensor. For video, a 3D or causal VAE compresses time as well as space, so that a few seconds of footage becomes a manageable block of numbers.
2. Patchify. The latent is cut into small patches, in video spanning a few frames as well as a small spatial area ("spacetime patches"), and each patch becomes a token with a position embedding that says where and when it sits.
3. Transformer blocks. Stacks of attention and feed-forward layers process all the tokens together. Conditioning enters in two ways: the noise level (timestep) usually modulates each block's normalization, and the text prompt enters through cross-attention or, in newer designs, by joining the text tokens and image tokens into one sequence.
4. Unpatchify and decode. The output tokens are reassembled into a latent, and after the last denoising step the VAE decodes it back into frames.

Many recent models also replace classic diffusion's noise schedule with flow matching or rectified flow, training the network to predict a straight path from noise to data. The transformer backbone is the same; the training objective is simpler and usually needs fewer sampling steps.

Why It Took Over Video

Three properties made DiTs the default for video generation:

• Scaling. Transformer quality improves smoothly as parameters, data and compute grow, the same scaling behavior seen in language models, and the engineering for training large transformers was already mature.
• One treatment of space and time. Spacetime patches let attention relate any part of any frame to any other, which is what keeps objects and characters consistent across a shot. Variable resolutions, aspect ratios and durations become different sequence lengths rather than different architectures.
• Multimodal sequences. Text, reference images, audio and video can all be represented as tokens in one sequence, which is how 2025 and 2026 models added native audio, multiple reference images and editing of existing footage.

The open models show the pattern most clearly, since commercial vendors publish few architectural details. Meta's Movie Gen paper (October 2024) described a 30-billion-parameter transformer trained with flow matching, generating 16 seconds of video at 16 frames per second. Tencent's HunyuanVideo (December 2024) had more than 13 billion parameters, the largest open video model at its release; HunyuanVideo 1.5 (November 2025) cut that to 8.3 billion, with a 3D causal VAE and sparse attention. Alibaba's Wan 2.1 (February 2025, Apache-2.0) paired a DiT trained with flow matching with a 3D causal VAE, and Wan 2.2 (July 2025) split denoising between two experts, one for the early, noisy steps that set layout and one for the late steps that add detail, for 27 billion parameters in total but 14 billion active per step. Lightricks' LTX-Video (December 2024) showed that a DiT with an aggressively compressing VAE could run faster than real time, producing 5 seconds of 24 fps video in about 2 seconds on an NVIDIA H100.

Costs and Open Problems

• Attention is quadratic. Doubling the number of tokens, from a longer clip or a higher resolution, roughly quadruples the cost of full attention. In one 2025 measurement, attention accounted for 800 of the 945 seconds HunyuanVideo needed to generate a five-second 720p clip, which is why sparse and windowed attention, such as Sliding Tile Attention, is an active research area.
• Many steps. Each denoising step runs the whole transformer. Distillation and few-step samplers cut the count, but real-time generation remains hard at high quality.
• Length and causality. A standard DiT denoises a whole clip at once, so it can't stream, and extending a video means generating new clips conditioned on old ones. Research since late 2024 has pushed DiTs toward causal, frame-by-frame generation that borrows from autoregressive models: CausVid (December 2024) converted a bidirectional video DiT into one that streams frames after a short delay, Self Forcing (June 2025) trained on the model's own rollouts to stream in real time on a single GPU, and Sand AI's MAGI-1 (May 2025) denoises video in 24-frame chunks so that the cost per chunk stays constant as a video grows.
• Physics and consistency. Scale has improved coherence a great deal, but object permanence and physical plausibility still fail on complex motion, one motivation for research on world models.