score matching does: learns the gradient of the log-density without normalizing

denoising score matching does: learns to denoise, which equals learning the score

Denoising score matching learns to denoise by estimating the score (gradient of log probability) of data distributions

to standardize: when you need zero mean and unit variance for gradient-based optimization

Standardize when zero mean and unit variance are required for gradient-based optimization

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates

natural gradient descent does: preconditions with inverse Fisher matrix

Natural gradient descent optimizes using the Fisher information matrix's inverse as the metric

Proximal gradient methods for learning

Proximal gradient descent efficiently handles non-differentiable L1 regularization by combining gradient descent with a proximity operator

AdaGrad's learning rate decays to zero

AdaGrad adjusts learning rate by accumulating squared gradients, causing it to decay to zero as denominator grows exponentially