What is the formula for calculating the mutual information between two discrete random variables X and Y?

Mutual information I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)

Mutual information measures dependence between variables X and Y

Write the equation for cross-entropy loss

H(y, p) = -Σ(y_i * log(p_i)) for all i

Write the formula for covariance between X and Y

Cov(X, Y) = Σ((Xi - X̄)(Yi - Ȳ)) / (n - 1)

Shannon's channel capacity: C = B log₂(1 + S/N) bits per second

Shannon's formula: C = B log₂(1 + S/N) defines channel capacity in bits/s

Write the formula for Pearson correlation coefficient

r = Σ((xi - x̄)(yi - ȳ)) / (√Σ(xi - x̄)² * √Σ(yi - ȳ)²)

Write the formula for KL divergence D_KL(P||Q)

D_KL(P||Q) = Σ P(x) log(P(x)/Q(x)) for all x in the support of P