Mutual information formula: I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y)))
Mutual information
Mutual information formula: I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y)))
Mutual information quantifies the amount of information obtained about one random variable by observing another. It is calculated using the joint probability distribution p(x,y) and the marginal distributions p(x) and p(y). The formula I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y))) captures this relationship by summing over all possible values of X and Y.
The formula I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y))) uses the joint probability distribution p(x,y) and the marginal distributions p(x) and p(y) to measure the mutual dependence between X and Y. This formula is more general than the correlation coefficient and determines how different the joint distribution of (X,Y) is from the product of the marginal distributions of X and Y.
The concept of mutual information is linked to entropy, which quantifies the expected amount of information held in a random variable. Mutual information measures the expected value of the pointwise mutual information (PMI), which is the amount of information obtained about one random variable by observing another. This concept was defined and analyzed by Claude Shannon in his landmark paper "A Mathematical Theory of Communication."
Example
Consider two random variables X and Y with joint distribution p(x,y) and marginal distributions p(x) and p(y). If X and Y are independent, then p(x,y) = p(x)p(y). In this case, the mutual information I(X;Y) = ∑_x∈X ∑_y∈Y p(x,y) log(p(x,y)/(p(x)p(y))) will be zero, indicating no mutual dependence between X and Y.
Understanding the formula for mutual information is crucial for analyzing the dependence between random variables and quantifying the amount of information shared between them. This knowledge is fundamental in fields such as information theory, communication systems, and machine learning.
Related concepts
Entropy (information theory)
H(X) = −∑x∈X p(x) log(p(x))
Cross-entropy
Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))
Kullback–Leibler divergence
Kullback–Leibler divergence formula: D_KL(P||Q) = ∑_x∈X P(x) log(P(x)/Q(x))
Expected value
Expected value formula: E[X] = Σ [x * P(x)]
Perplexity
Perplexity = 2^H
Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits
Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system
One email a day: 5 concepts + the 5 stories that matter →
Swipe through 100 ML concepts daily
Open TickerNews