cross-entropy equals negative log-likelihood for classification

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Cross-entropy H(p,q) = -Σ p(x) log q(x) quantifies approximation quality between distributions p and q

log-loss / cross-entropy loss penalizes: confident wrong predictions more heavily

Log-loss penalizes confident incorrect predictions more heavily

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system

Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

soft targets carry more information than hard labels: they encode class similarities

Soft targets carry more information than hard labels because they encode class similarities

temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform

As T approaches 0, softmax concentrates probabilities; as T approaches ∞, probabilities become uniform