data augmentation does for generalization: artificially expands training set

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

dropout works as regularization: it approximates an ensemble of subnetworks

Dropout randomly deactivates neurons during training, simulating an ensemble of subnetworks, thus preventing co-adaptation and improving generalization

Adam vs SGD: Adam adapts per-parameter rates, SGD often generalizes better with tuning

Adam adjusts learning rates per-parameter, SGD generalizes better with tuning

gradient accumulation simulates larger batch sizes without more memory

Gradient accumulation reduces memory usage by dividing a large batch into smaller mini-batches, accumulating gradients before updating model weights

Prompt engineering

The GenAI model learns tasks from examples in the prompt

the Gram-Schmidt process does: orthogonalizes a set of vectors

Orthogonalizes a set of vectors using Gram-Schmidt