ELBO formula in variational inference
Variational autoencoder
ELBO formula in variational inference
The ELBO (Evidence Lower Bound) formula is central to variational inference as it provides a lower bound for the log-likelihood of the observed data. This bound is used to optimize the parameters of the variational distribution.
The ELBO formula is expressed as: ELBO = E_q[log p(X|Z)] - KL(q(Z)||p(Z)), where E_q denotes the expectation with respect to the variational distribution q(Z), log p(X|Z) is the log-likelihood of the data X given the latent variable Z, and KL(q(Z)||p(Z)) is the Kullback-Leibler divergence between the variational distribution q(Z) and the prior distribution p(Z).
Understanding and calculating the ELBO is crucial for training variational models like VAEs. By maximizing the ELBO, one can indirectly maximize the log-likelihood of the observed data while also ensuring that the variational distribution stays close to the prior distribution.
Example
In a VAE, suppose X represents images and Z represents latent variables. The ELBO can be computed by taking the expectation of the log-likelihood of the images given the latent variables and subtracting the KL divergence between the variational distribution of the latent variables and the prior distribution.
Maximizing the ELBO helps in learning meaningful latent representations and improves the generalization of the model.
Related concepts
t-SNE preserves local structure
t-SNE preserves local structure by converting distances to probabilities and minimizing Kullback-Leibler divergence
Expectation–maximization algorithm
EM algorithm iteratively maximizes likelihood estimates with latent variables
Write the Bellman equation for reinforcement learning
Bellman equation: V(s) = max_a [R(s,a) + γ Σ P(s'|s,a) V(s')]
Stochastic gradient descent
Policy Gradient Theorem Equation
the do-calculus does: computes interventional probabilities from observational data
Do-calculus computes interventional probabilities from observational data
Metropolis–Hastings algorithm
Metropolis-Hastings algorithm samples from difficult distributions
One email a day: 5 concepts + the 5 stories that matter →
Swipe through 100 ML concepts daily
Open TickerNews