Rate-distortion theory: minimum bits to represent data within distortion D

Shannon's source coding theorem: you can't compress below entropy

Shannon's theorem: Data compression can't exceed entropy limit

Huffman coding

Huffman coding is an entropy-optimal prefix code for lossless data compression

LoRA (machine learning)

LoRA uses r << d for efficient adaptation

Randomized algorithm

Randomized algorithms use random bits for expected polynomial time

non-convex loss landscapes are hard: many local minima and saddle points

Non-convex loss landscapes are hard due to many local minima and saddle points

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Log-loss penalizes confident incorrect predictions more heavily