Entropy (information theory)

H(X) = −∑x∈X p(x) log(p(x))

Information entropy quantifies the average uncertainty or information content associated with a random variable's potential states. It measures the expected amount of information needed to describe the variable's state, considering the probabilities of each state. The formula H(X) = −∑x∈X p(x) log(p(x)) captures this concept by summing the product of the probability of each state and the logarithm of that probability.

Example

For a fair coin (X = {Heads, Tails}), the probabilities are p(Heads) = 0.5 and p(Tails) = 0.5. Using the entropy formula, H(X) = −(0.5 log(0.5) + 0.5 log(0.5)) = 1 bit. This means that, on average, 1 bit of information is needed to describe the outcome of a fair coin toss.

Understanding the formula for information entropy is crucial in fields like data compression, cryptography, and communication systems, as it helps quantify the amount of uncertainty or information in a given set of outcomes.

Related concepts

Mutual information

Mutual information formula: I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y)))

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system

Cross-entropy

Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Cross-entropy H(p,q) = -Σ p(x) log q(x) quantifies approximation quality between distributions p and q

Perplexity

Perplexity = 2^H

Expected value

Expected value formula: E[X] = Σ [x * P(x)]

One email a day: 5 concepts + the 5 stories that matter →

Swipe through 100 ML concepts daily

Open TickerNews