# LoRA (Low-Rank Adaptation)

> LoRA (Low-Rank Adaptation) fine-tunes a model by training small low-rank matrices beside frozen weights, cutting memory, compute and storage.

Source: https://metavert.io/lora  
Published: 2026-10-07  
Updated: 2026-10-07

**LoRA (Low-Rank Adaptation)** is a [fine-tuning](https://metavert.io/fine-tuning) method that freezes a pretrained model's weights and trains only a pair of small low-rank matrices added alongside selected weight matrices, so that a task-specific update costs a tiny fraction of the parameters, memory and storage of retraining the whole network. It is the most widely used member of the [parameter-efficient fine-tuning](https://metavert.io/parameter-efficient-fine-tuning) family and the default way [open-weight models](https://metavert.io/open-weight-models) are customized.

### How the Method Works

The original paper (Hu et al., June 2021) observed that the change a model needs in order to adapt to a new task can be approximated by a low-rank matrix. Instead of updating a weight matrix *W* directly, LoRA learns two narrow matrices *A* and *B* whose product *BA* is added to *W*. The authors reported that, relative to fully fine-tuning GPT-3 175B, this cut trainable parameters by a factor of 10,000 and GPU memory by a factor of three, while matching or exceeding full fine-tuning quality on RoBERTa, DeBERTa, GPT-2 and GPT-3. Because *BA* can be folded back into *W* after training, a merged LoRA adds no inference latency, which earlier bottleneck adapters could not claim.

Three settings do most of the work, as documented in the Hugging Face PEFT library:

| Setting | Plain meaning | Practical effect |
| --- | --- | --- |
| Rank (`r`) | Width of the two update matrices | Lower rank means fewer trainable parameters and less capacity to learn something new |
| Alpha (`lora_alpha`) | Scaling factor on the update, applied as alpha divided by rank | Controls how loudly the adapter speaks relative to the frozen weights |
| Target modules | Which layers receive an adapter | PEFT's default is the query and value projections of each attention block; `all-linear` covers every linear layer |

### QLoRA and Consumer Hardware

QLoRA (Dettmers et al., May 2023) combined LoRA with [quantization](https://metavert.io/model-quantization): the frozen base model is loaded in 4-bit precision and gradients flow through it into full-precision adapters. The paper introduced a 4-bit NormalFloat data type, double quantization and paged optimizers, and reported fine-tuning a 65-billion-parameter model on a single 48 GB GPU while preserving the task performance of full 16-bit fine-tuning. That result is what moved fine-tuning from data-center clusters to a single workstation card; the PEFT project's own examples include tuning a 7-billion-parameter Llama 2 model with QLoRA on a 16 GB GPU. The trade is speed and some numerical care, since the base model is being read through a lossy representation.

### Variants Worth Knowing

**rsLoRA** (Kalajdzievski, November 2023) argues that dividing by the rank slows learning at higher ranks and proves the scale should instead be alpha over the square root of the rank, which makes large ranks worth their extra compute. **DoRA** (Liu et al., February 2024; ICML 2024 oral) splits each weight into magnitude and direction, applies LoRA only to direction, and reports consistent gains over LoRA on LLaMA, LLaVA and VL-BART; the PEFT documentation notes that it helps most at low ranks and carries more overhead until merged. Both are single switches in PEFT (`use_rslora`, `use_dora`), alongside a long list of alternative initializations. Most of these variants are supported by one or a few papers each, and none has displaced plain LoRA as the baseline.

### Adapters as Artifacts

A LoRA is a file, usually megabytes against a base model's gigabytes: the PEFT repository cites a 19 MB adapter checkpoint for a model whose full weights occupy 11 GB. That changes how models are shipped. An adapter can be *merged* into the base to produce a standalone model, kept separate and *switched* at runtime, mixed within one batch so different requests use different adapters, or blended with other adapters by weighted combination. Inference servers treat them as first-class objects: [vLLM](https://metavert.io/vllm) lets a request name an adapter as if it were a model and, when enabled, loads and unloads adapters through API endpoints without a restart. One base model in memory can therefore serve many specializations, and an adapter is tied to the exact base it was trained on.

### Limits Against Full Fine-Tuning

LoRA is not free. Biderman et al. (May 2024) compared the two on programming and mathematics and found that, at standard low ranks, LoRA substantially underperformed full fine-tuning, and that full fine-tuning learned weight changes with a rank 10 to 100 times greater than typical LoRA configurations. The same study found LoRA forgot less of the base model's abilities outside the target domain. That is one paper on two domains, but it matches the practical rule: LoRA is strong for style, format and narrow-task adaptation, and weaker when a model must absorb a large body of new knowledge or skill. The [LoRA versus full fine-tuning](https://metavert.io/compare/lora-vs-full-fine-tuning) comparison covers the decision in detail.

As of October 2026 the reference implementation is Hugging Face [PEFT](https://metavert.io/hugging-face) (Apache 2.0), which integrates with Transformers and Diffusers. Unsloth, an open-source training framework with an Apache 2.0 core, packages LoRA and QLoRA together with full fine-tuning, DPO and [GRPO](https://metavert.io/grpo) and exports to formats such as GGUF; its claims of twice the training speed and 70% less VRAM are vendor-reported.

## Related Topics

- [Parameter-Efficient Fine-Tuning](https://metavert.io/parameter-efficient-fine-tuning) — The family of methods LoRA belongs to
- [Fine-Tuning](https://metavert.io/fine-tuning) — The general technique LoRA makes cheaper
- [LoRA vs Full Fine-Tuning](https://metavert.io/compare/lora-vs-full-fine-tuning) — When the low-rank shortcut is and is not enough
- [Model Quantization](https://metavert.io/model-quantization) — The other half of QLoRA
- [Model Merging](https://metavert.io/model-merging) — Combining adapters and fine-tunes in weight space
- [Open-Weight Models](https://metavert.io/open-weight-models) — The models LoRA adapters are built on
- [vLLM](https://metavert.io/vllm) — Serves many adapters from one base model
- [Hugging Face](https://metavert.io/hugging-face) — Home of the PEFT library and most public adapters
- [Post-Training](https://metavert.io/post-training) — The broader stage in which adapters are trained

## Further Reading

- [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685) — Hu et al., arXiv, June 2021
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314) — Dettmers et al., arXiv, May 2023
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://arxiv.org/abs/2312.03732) — Kalajdzievski, arXiv, November 2023
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://arxiv.org/abs/2402.09353) — Liu et al., arXiv, February 2024
- [LoRA Learns Less and Forgets Less](https://arxiv.org/abs/2405.09673) — Biderman et al., arXiv, May 2024
- [LoRA conceptual guide](https://huggingface.co/docs/peft/main/en/conceptual_guides/lora) — Hugging Face PEFT documentation, accessed October 2026
- [LoRA developer guide (rsLoRA, DoRA, merging, multiple adapters)](https://huggingface.co/docs/peft/developer_guides/lora) — Hugging Face PEFT documentation, accessed October 2026
- [PEFT repository](https://github.com/huggingface/peft) — Hugging Face on GitHub, accessed October 2026
- [LoRA Adapters](https://docs.vllm.ai/en/latest/features/lora.html) — vLLM documentation, accessed October 2026
- [Unsloth documentation](https://unsloth.ai/docs) — Unsloth, accessed October 2026