Definition
Plain language
A cheap way to fine-tune a big model by training small added pieces instead of all its weights.
As stated in the literature
Low-Rank Adaptation, a parameter-efficient fine-tuning method that trains small low-rank update matrices while keeping the base weights frozen.
Why it matters: It made domain-specific fine-tuning of large models feasible for groups without huge GPU budgets.
For example, instead of updating all 70 billion parameters of a base model, LoRA might train a few million parameters in small inserted matrices, getting most of the benefit at a fraction of the cost.
Heard on the show
“The controlled experiment is on relatively small models, with rank-sixteen LoRA adapters, on a thousand preference pairs.”Episode 004 — The Sycophancy Circuit That Survives Alignment Training