LoRA vs Full Fine-Tuning

Why LoRA is cheap

Hu et al. (Microsoft, June 2021) proposed that the change in weights during adaptation has low intrinsic rank, and so can be represented as the product of two thin matrices. Only those matrices receive gradients. Because the optimizer no longer stores state for the frozen weights, the paper reports a reduction in training VRAM on GPT-3 175B from 1.2 TB to 350 GB, a 10,000-fold reduction in checkpoint size at rank 4 with only the query and value projections adapted, and a 25% training speed-up on that model. After training, the product can be added back into the base weights, so a merged LoRA model runs with no additional inference latency.

QLoRA (Dettmers et al., May 2023) extended the saving by back-propagating through a frozen base model quantized to 4 bits (see model quantization). The authors report fine-tuning a 65B model on one 48 GB GPU, where 16-bit full fine-tuning would need more than 780 GB. Their direct comparison with full fine-tuning covers models up to 3B parameters; at 33B and 65B the full fine-tuning baseline was too expensive to run, so equivalence at that scale is argued rather than measured.

Does it match full fine-tuning?

The original paper says yes, on GLUE and on generation tasks such as WikiSQL and SAMSum. Those are modest shifts for the models involved. Biderman et al. ("LoRA Learns Less and Forgets Less", TMLR 2024) tested harder shifts on Llama-2-7B, in programming and mathematics, under both instruction tuning (about 100K examples) and continued pretraining (20B tokens). Their conclusion: "in the standard low-rank settings, LoRA substantially underperforms full finetuning".

The detail matters. For code instruction tuning, rank 16 reached 0.358 on HumanEval against 0.497 for full fine-tuning, while rank 256 reached 0.498 and closed the gap. For continued pretraining on code the gap persisted at every rank tested (0.224 vs 0.263). For mathematics instruction tuning, a smaller shift from the base model, the gap was small (0.634 vs 0.642 on GSM8K at rank 256). These results come from one model family and two domains. QLoRA's authors made a related observation: adapting only the query and value matrices was not enough to match full fine-tuning on larger models, and adapters on all linear layers were needed.

Forgetting and structural differences

The same constraint that limits learning also protects the base model. Biderman et al. found LoRA retained more performance outside the target domain than full fine-tuning and more than standard regularizers, and kept generations more diverse. For teams that want a specialist that still behaves like the base model elsewhere, that is a benefit and not just a consolation.

Shuttleworth et al. ("An Illusion of Equivalence", 2024–2025) add a caution. Even where task accuracy matches, LoRA-trained weights contain new high-ranking singular vectors, which the authors call intruder dimensions, that are absent after full fine-tuning. They link these to forgetting and report that models fine-tuned sequentially with LoRA accumulate them and tend to perform worse in continual learning settings. Equal benchmark scores do not imply equal models.

Tuning and speed in practice

LoRA has its own sensitivities. Biderman et al. report that it needs learning rates about an order of magnitude higher than full fine-tuning (roughly 5e-5 to 5e-4), that setting alpha to twice the rank matters at high ranks, and that targeting all modules, MLP blocks included, beats attention-only adapters. They also note that in standard implementations LoRA tended to train more slowly than full fine-tuning at a fixed batch size, which contrasts with the 25% speed-up reported for GPT-3 in the original paper. The memory saving is consistent across sources; the wall-clock saving is not.