LoRA vs full fine-tuning: LoRA trains rank-r adapters (~0.1% params), full FT updates everything

2024 in hip-hop

LoRA rank r controls model capacity and parameters

LoRA (machine learning)

LoRA uses r << d for efficient adaptation

QLoRA adds

QLoRA quantizes base model to 4-bit, trains LoRA adapters in FP16

Alex Lora Cercos

Alex Lora is a Spanish film director

instruction tuning does: fine-tunes on (instruction, response) pairs

Fine-tunes on (instruction, response) pairs

L1 vs L2 regularization: L1 gives sparsity (feature selection), L2 gives small weights

L1 regularization: L1 = L2 + sparsity; L2 regularization: L2 = L1 + small weights