KL divergence is always ≥ 0 and equals 0 only when P = Q exactly

Kullback–Leibler divergence

Kullback–Leibler divergence formula: D_KL(P||Q) = ∑_x∈X P(x) log(P(x)/Q(x))

Cross-entropy H(p,q) = -Σ p(x) log q(x) measures how well q approximates p

Cross-entropy H(p,q) = -Σ p(x) log q(x) quantifies approximation quality between distributions p and q

Jensen–Shannon divergence

Jensen-Shannon divergence formula: D_JS(P||Q) = 1/2 * D_KL(P||(M)) + 1/2 * D_KL(Q||(M))

Lyapunov exponents measure: rate of divergence of nearby trajectories in a dynamical system

Lyapunov exponents measure the rate of divergence of nearby trajectories in a dynamical system

Entropy H = -Σ p(x) log₂ p(x) measures average surprise in bits

Entropy H = -Σ p(x) log₂ p(x) quantifies uncertainty in a system

Chebyshev's inequality says: P(|X-μ| ≥ kσ) ≤ 1/k²

Chebyshev's inequality states: P(|X-μ| ≥ kσ) ≤ 1/k²