Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

Cross-entropy loss is a measure used in machine learning to quantify the difference between two probability distributions. It is particularly useful for comparing the predicted probabilities (q) with the actual probabilities (p) of events.

Cross-entropy loss helps in optimizing machine learning models by providing a clear metric for how well the model's predictions align with the true data distribution. By minimizing this loss, models can improve their accuracy and performance over time.

The equation for cross-entropy loss is H(p, q) = -Σ(p(x) * log(q(x))), where p(x) represents the actual probability distribution, q(x) represents the predicted probability distribution, and the summation Σ extends over all possible events x.

Example

Suppose we have a binary classification problem with two classes: A and B. Let p(A) = 0.7 and p(B) = 0.3 be the true probabilities. If our model predicts q(A) = 0.6 and q(B) = 0.4, the cross-entropy loss would be calculated as follows: H(p, q) = -[ (0.7 * log(0.6)) + (0.3 * log(0.4)) ]

Understanding and applying the cross-entropy loss equation is crucial for improving the performance of machine learning models, as it directly influences how well the model's predictions match the actual data.

Related concepts

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Cross-entropy H(p,q) = -Σ p(x) log q(x) quantifies approximation quality between distributions p and q

Entropy (information theory)

H(X) = −∑x∈X p(x) log(p(x))

Perplexity

Perplexity = 2^H

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system

Kullback–Leibler divergence

Kullback–Leibler divergence formula: D_KL(P||Q) = ∑_x∈X P(x) log(P(x)/Q(x))

Mutual information

Mutual information formula: I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y)))

One email a day: 5 concepts + the 5 stories that matter →

Swipe through 100 ML concepts daily

Open TickerNews