Why non-convex loss landscapes are hard: many local minima and saddle points

Why proximal gradient descent is needed for L1 optimization

Proximal gradient descent handles non-differentiable L1 regularization, enabling sparse solutions

Why SGD with momentum escapes local minima better than vanilla SGD

Momentum SGD accumulates velocity, helping to overcome shallow local minima

How does the concept of convexity in optimization relate to finding the global minimum in a non-linear cost function?

Convexity ensures a single global minimum in non-linear cost functions

Why L1 regularization produces sparse solutions — the diamond corners touch axes

L1 regularization promotes sparsity by penalizing non-zero coefficients, effectively driving some to zero

Why second-order methods (Newton's) converge faster but are expensive: O(n³) per step

Newton's method has quadratic convergence but requires cubic computational cost per iteration

Why the curse of dimensionality makes nearest neighbor search unreliable

High-dimensional spaces increase distance ambiguity, reducing nearest neighbor search reliability