Low-Rank Adaptation
Also called LoRA, LoRA adapter, QLoRA, Parameter-efficient fine-tuning
A cheap way to fine-tune a model by training a small set of extra weights while leaving the original model unchanged.
What is LoRA?
LoRA (low-rank adaptation) is a technique for fine-tuning an AI model without retraining all of it. The original model's weights are frozen, and a small number of new weights, called an adapter, are trained on your examples. When the model runs, the adapter nudges its behaviour towards your task.
It's widely used for language models and for image models, where LoRAs teach a diffusion model a particular style, character or product.
How does it work?
Fine-tuning normally adjusts huge grids of numbers inside the model. LoRA represents the change to each grid as the product of two much thinner grids, which together have only a tiny fraction of the numbers. Only those thin grids are trained. The resulting adapter file is often megabytes rather than gigabytes, and it can be loaded on top of the base model, swapped for another, or merged into it.
QLoRA goes further by keeping the frozen base model in quantized form during training, so large models can be fine-tuned on a single graphics card.
What is it commonly confused with?
- Full fine-tuning. Full fine-tuning updates every weight. LoRA updates a small add-on, which is much cheaper and often reaches similar quality for narrow tasks.
- A new model. A LoRA adapter only works with the base model it was trained on.
Why does it matter?
LoRA made fine-tuning affordable for small teams and individuals, and lets one base model serve many tasks with a different small adapter for each. Many fine-tuning services and image tools use it behind the scenes. As with any fine-tuning, it's best for shaping behaviour and style, not for teaching facts that change.
