Proximal gradient methods for learning

Regularization (mathematics)

L1 regularization results in sparse solutions

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

to standardize: when you need zero mean and unit variance for gradient-based optimization

Standardize when zero mean and unit variance are required for gradient-based optimization

natural gradient descent does: preconditions with inverse Fisher matrix

Natural gradient descent optimizes using the Fisher information matrix's inverse as the metric

gradient accumulation simulates larger batch sizes without more memory

Gradient accumulation reduces memory usage by dividing a large batch into smaller mini-batches, accumulating gradients before updating model weights

Convex optimization

Convex functions have only one global minimum