LoRA

LoRA, low-rank adaptation, fine-tunes a small set of adapter weights instead of updating every parameter of a base model.

LoRA injects trainable low-rank matrices into attention or MLP layers while freezing the base checkpoint. That cuts memory and makes it practical to keep many task adapters. QLoRA quantizes the base model during training to reduce GPU RAM further. Adapter quality still depends on data, learning rate, and evals. LoRA does not change the base license by itself; check both the base weights and any a...