Shannon's source coding theorem: you can't compress below entropy

Huffman coding

Huffman coding is an entropy-optimal prefix code for lossless data compression

Rate-distortion theory: minimum bits to represent data within distortion D

Rate-distortion theory: minimum bits to represent data within distortion D = R(D)

temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform

As T approaches 0, softmax concentrates probabilities; as T approaches ∞, probabilities become uniform

GPTQ quantization does

Post-training quantization using second-order information for model compression

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system

Kolmogorov complexity

Kolmogorov complexity is uncomputable