vLLM

vLLM is an open-source library for fast and memory-efficient LLM inference and serving. Originally developed at UC Berkeley, vLLM introduced the PagedAttention algorithm, which revolutionized how LLM serving manages GPU memory by applying virtual memory concepts to the attention key-value cache.

PagedAttention allows vLLM to serve more concurrent requests per GPU than previous approaches, dramatically improving throughput and reducing the cost of LLM inference. The library has been widely adopted by AI companies and cloud providers as the foundation for their model serving infrastructure.

In the model-building toolchain layer, vLLM has become essential infrastructure for the agentic economy — the serving engine that makes it practical to run the LLMs powering AI agents at scale, with the throughput and cost efficiency required for production deployment.

What It Is

The project's documentation describes vLLM as "a fast and easy-to-use library for LLM inference and serving" that began in the Sky Computing Lab at UC Berkeley and is now maintained by a community of more than 2,000 contributors. It is Apache-2.0 licensed; the GitHub repository showed about 93,000 stars when checked on 6 October 2026, and the latest release listed was v0.31.0. The PagedAttention paper appeared at the ACM Symposium on Operating Systems Principles in 2023. vLLM is not a model or a training framework: it loads weights — typically open-weight models from Hugging Face — and turns them into a high-throughput endpoint. The documentation lists support for more than 200 model architectures on NVIDIA and AMD GPUs and several CPU families.

Continuous Batching and Throughput

Static batching waits for a batch to fill and then holds every slot until the longest sequence finishes. Continuous batching, which vLLM lists among its core features, instead admits new requests into the running batch as others complete, so the GPU stays occupied. PagedAttention is what makes that practical: because the key-value cache is allocated in blocks rather than as one contiguous region per request, memory freed by a finished request can be reused immediately. The documentation lists related optimizations: chunked prefill, prefix caching, speculative decoding and quantized formats. Its "state-of-the-art serving throughput" wording is a developer claim, and gains depend on workload shape.

The OpenAI-Compatible Server

Most deployments use vLLM through its HTTP server, started with vllm serve <model>. The server implements the OpenAI Completions, Chat Completions, Responses and Embeddings APIs, among others, and the documentation also lists an Anthropic Messages API and gRPC support. Client code written against a commercial API can therefore be pointed at a self-hosted model by changing a base URL. Chat formatting comes from the model's chat template, which can be supplied with --chat-template when a model ships without one.

Adapter Serving

vLLM can serve many LoRA adapters on one copy of a base model. Adapters are registered at start-up with --enable-lora and --lora-modules, and a request selects one by passing its name in the model field, "as if it were any other model" in the documentation's words. With runtime updating enabled, adapters can be loaded and unloaded over HTTP (/v1/load_lora_adapter, /v1/unload_lora_adapter), and a load_inplace option replaces an adapter under the same name. Flags such as --max-loras and --max-lora-rank bound memory use.

Role in RL Training Loops

Online reinforcement fine-tuning methods such as GRPO improve a model using completions the model itself generates during training, so sampling speed bounds training speed. Hugging Face's TRL documentation states that "generation is often the main bottleneck when training with online methods" and integrates vLLM for that step in two modes: a separate vLLM server that the trainer calls, or a co-located instance sharing the training GPUs, which avoids a second server at the cost of possible resource contention. In-place adapter reloading serves the same pattern in custom loops, where the sampler must pick up newly trained adapter weights between steps.

Reproducibility

The vLLM reproducibility page documents a seed parameter that controls the engine's random number generators. It defaults to 0, which the docs say keeps results consistent from run to run even at non-zero temperature. Two caveats are stated: results are only reproducible on the same hardware and the same vLLM version, and in online serving, where scheduling varies with load, reproducibility additionally requires the batch-invariance feature.