Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system

Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

cross-entropy equals negative log-likelihood for classification

Cross-entropy measures the difference between predicted probabilities and true labels, thus it equals negative log-likelihood, reflecting the cost of incorrect predictions

Entropy (information theory)

H(X) = −∑x∈X p(x) log(p(x))

Perplexity

Perplexity = 2^H

temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform

As T approaches 0, softmax concentrates probabilities; as T approaches ∞, probabilities become uniform