Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Cross-entropy H(p,q) = -Σ p(x) log q(x) quantifies approximation quality between distributions p and q

Entropy (information theory)

H(X) = −∑x∈X p(x) log(p(x))

A fair die has entropy of log₂(6) ≈ 2.58 bits

A fair die's entropy: log₂(6) ≈ 2.58 bits

temperature T in softmax(x/T) controls entropy: T→0 is argmax, T→∞ is uniform

As T approaches 0, softmax concentrates probabilities; as T approaches ∞, probabilities become uniform

Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

cross-entropy equals negative log-likelihood for classification

Cross-entropy measures the difference between predicted probabilities and true labels, thus it equals negative log-likelihood, reflecting the cost of incorrect predictions