Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))
Image: Edward Larsson, Public domain, via Wikimedia Commons
Jensen–Shannon divergence
Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))
The Jensen-Shannon divergence formula is a symmetric measure of similarity between two probability distributions P and Q. It is derived from the Kullback-Leibler divergence by averaging the two distributions (M = 1/2 * (P + Q)) and then calculating the divergence from each distribution to the average.
The formula D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M)) ensures that the divergence is always finite and symmetric, unlike the Kullback-Leibler divergence. This makes it a more robust measure for comparing probability distributions.
The Jensen-Shannon distance, which is the square root of the Jensen-Shannon divergence, provides a metric for quantifying the similarity between distributions. A smaller Jensen-Shannon distance indicates greater similarity between the distributions.
Example
Consider two probability distributions P and Q, where P = [0.1, 0.9] and Q = [0.8, 0.2]. The average distribution M = [0.45, 0.65]. The Jensen-Shannon divergence D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M)) can be calculated using the Kullback-Leibler divergence formula.
Understanding the Jensen-Shannon divergence formula is crucial for accurately measuring the similarity between probability distributions in various fields such as machine learning, information theory, and statistics.
Related concepts
Kullback–Leibler divergence
Kullback–Leibler divergence formula: D_KL(P||Q) = ∑_x∈X P(x) log(P(x)/Q(x))
Cross-entropy
Cross-entropy loss equation: H(p, q) = -Σ(p(x) * log(q(x)))
Discrete Fourier transform
Discrete Fourier Transform (DFT) equation: X[k] = Σ(n=0 to N-1) x[n] * e^(-j*2π*k*n/N)
Expected value
Expected value formula: E[X] = Σ [x * P(x)]
Entropy (information theory)
H(X) = −∑x∈X p(x) log(p(x))
Perplexity
Perplexity = 2^H
One email a day: 5 concepts + the 5 stories that matter →
Swipe through 100 ML concepts daily
Open TickerNews