ML//training//fine-tuning//LoRA

2023-04-18

LoRA (Low-Rank Adaptation) is a fine-tuning method that freezes a model's base weights and trains a small low-rank correction (two thin matrices whose product has rank 4 to 64) beside chosen weight matrices. Only the correction is trained, often well under 1% of the parameters, and it can be merged into the weights afterwards, so a foundation model can be customised on one GPU and served at no extra cost. The original paper matched full fine-tuning of GPT-3 175B while training about 10,000 times fewer parameters ([LoRA, Hu et al., 2021](https://arxiv.org/abs/2106.09685)).

For one weight matrix WWW of size d×kd\times kd×k, LoRA learns a correction of rank rrr much smaller than either side:

W′=W+BA,B∈Rd×r, A∈Rr×kW' = W + BA,\qquad B\in\mathbb{R}^{d\times r},\; A\in\mathbb{R}^{r\times k}W′=W+BA,B∈Rd×r,A∈Rr×k

WWW stays frozen, only BBB and AAA train, and BBB starts at zero so the adapted model begins exactly as the base. With d=k=4096d=k=4096d=k=4096 and r=8r=8r=8 the correction has 65,536 numbers against 16.8 million in WWW.

LoRA is a technique and SFT an objective, so they are not alternatives: SFT (or a preference method from post-training) says what to learn, LoRA says which parameters may change. Most SFT runs outside the big labs are SFT through LoRA.

Where to put it is a choice. The original paper adapted only WQW_QWQ and WVW_VWV, the attention matrices, and that became a habit; the QLoRA authors found that matching full fine-tuning needed it on every linear layer of the block, MLP included (Dettmers et al., 2023), which is the common default now.

QLoRA adds quantization on top, enough to fine-tune a 65B model on a single 48 GB GPU.

The tokenizer is almost never touched. Changing it means changing the embedding matrix, basically starting over.