Probabilistic ML · Interactive Explainer

Forward vs. Reverse KL Divergence

Swapping the two arguments of Kullback–Leibler divergence turns a mass-covering objective into a zero-forcing one. Here is the exact geometry behind why unimodal approximations stretch across empty valleys, when mode-seeking actually kicks in, and how this asymmetry governs diffusion models, distillation, and RLHF.

Oct 2026 Interactive note 10 min read

Suppose we have a complex target distribution $p(x)$, such as a multimodal distribution of natural images, text trajectories, or a Bayesian posterior, and we want to approximate it using a simpler parameterized distribution $q_\phi(x)$.

At any point $x$, the log-ratio $\log \frac{p(x)}{q_\phi(x)}$ measures how surprised one distribution is by the other, and the Kullback–Leibler (KL) divergence averages that mismatch across all $x$ (measured in nats when using the natural logarithm $\ln$). Because KL divergence weights that average by whichever distribution is written first, it is asymmetric ($\mathrm{D}_{\mathrm{KL}}(p \parallel q_\phi) \neq \mathrm{D}_{\mathrm{KL}}(q_\phi \parallel p)$), forcing a fundamental choice:

Forward / Inclusive KL

$\displaystyle \min_\phi \; \mathrm{D}_{\mathrm{KL}}(p \parallel q_\phi)$

$$\mathrm{D}_{\mathrm{KL}}(p \parallel q_\phi) = \mathbb{E}_{x \sim p}\!\left[\log \frac{p(x)}{q_\phi(x)}\right]$$

Expectation under target $p(x)$: Evaluated wherever data actually lives.
Infinite penalty if $p(x) > 0$ and $q_\phi(x) \to 0$: Forbids $q_\phi$ from leaving any mode of $p$ uncovered.
Behavior: Mass-covering (mean-seeking). Stretches $q_\phi$ across all modes, even if that forces probability mass into empty valleys.
Where it appears: Maximum Likelihood Estimation (MLE), autoregressive pretraining, diffusion & flow matching, Expectation Propagation.

Reverse / Exclusive KL

$\displaystyle \min_\phi \; \mathrm{D}_{\mathrm{KL}}(q_\phi \parallel p)$

$$\mathrm{D}_{\mathrm{KL}}(q_\phi \parallel p) = \mathbb{E}_{x \sim q_\phi}\!\left[\log \frac{q_\phi(x)}{p(x)}\right]$$

Expectation under model $q_\phi(x)$: Evaluated wherever the model chooses to put mass.
Infinite penalty if $q_\phi(x) > 0$ and $p(x) \to 0$: Forces $q_\phi(x) = 0$ wherever $p(x) = 0$.
Behavior: Zero-forcing (and consequently mode-seeking). Concentrates onto a single high-density mode rather than bridging a zero-density valley.
Where it appears: Variational Inference (ELBO), Score Distillation Sampling (SDS / DMD), on-policy RLHF, policy gradients.

1. The Bimodal Gaussian Experiment

To see both objectives in action, let’s fit a single unimodal Gaussian $q_\phi(x) = \mathcal{N}(x; \mu_q, \sigma_q^2)$ to a two-component Gaussian mixture $p(x)$ with component weights $\pi_1$ and $\pi_2 = 1 - \pi_1$, means $\mu_1 = 0$ and $\mu_2 = \Delta\mu$, and component standard deviation $\sigma$:

$$p(x) = \pi_1\,\mathcal{N}(x; 0, \sigma^2) + \pi_2\,\mathcal{N}(x; \Delta\mu, \sigma^2), \qquad q_\phi(x) = \mathcal{N}(x; \mu_q, \sigma_q^2)$$

Starting with balanced weights ($\pi_1 = \pi_2 = \tfrac{1}{2}$) and unit width ($\sigma = 1$), watch how the optimal Forward KL fit $q_{\text{fwd}}^*(x)$ and Reverse KL fit $q_{\text{rev}}^*(x)$ evolve as $\Delta\mu$ increases across four distinct regimes:

Target mixture $p(x)$ Forward KL $\arg\min_q \mathrm{D}_{\mathrm{KL}}(p \parallel q)$ Reverse KL (global min) Reverse KL (mode-locked local min)

1 · $\Delta\mu = 0$ Unimodal target

When the two components coincide, $p(x)$ is a single Gaussian. Both Forward and Reverse KL recover $q^*(x) = p(x)$ with zero divergence.

2 · $\Delta\mu = 2.5$ Two peaks, shallow valley

$p(x)$ now has two peaks, yet Reverse KL is still centered at $\mu_q = 1.25$! Because the valley isn't near zero, Reverse KL does not seek a single mode yet.

3 · $\Delta\mu = 5.0$ Coexisting minima (pre-jump)

The centered solid red curve ($0.48\text{ nats}$) is still the global best, while the taller dotted red curve at $x \approx 5$ ($0.67\text{ nats}$) is the emerging mode-locked local minimum.

4 · $\Delta\mu = 8.5$ Deep zero-density valley

Past $\Delta\mu \approx 5.60\sigma$, the mode-locked curve wins globally: Reverse KL snaps onto one mode (peaking $2\times$ taller than $p$ since it packs $100\%$ mass into one peak).

Interactive Sandbox

Explore the Optimization Landscape

Presets:

Mode separation $\Delta\mu$ 8.0

Left mode weight $\pi_1$ 0.50

Component std $\sigma$ 1.00

Forward KL $\arg\min_q \mathrm{D}_{\mathrm{KL}}(p \parallel q)$ μ = 4.00, σ = 4.12 $\mathrm{D}_{\mathrm{KL}}(p \parallel q^*) = $ 0.724 nats

Reverse KL $\arg\min_q \mathrm{D}_{\mathrm{KL}}(q \parallel p)$ μ = 8.00, σ = 1.00 $\mathrm{D}_{\mathrm{KL}}(q^* \parallel p) = $ 0.693 nats

2. Forward KL: Why Isn’t $q(x) = 0$ in the Empty Valley?

A natural question arises when looking at the well-separated plot (e.g., $\Delta\mu = 8$ or $8.5$): if $p(x) \approx 0$ in the valley between the two modes, why does Forward KL place its highest probability density right in the middle of that empty valley at $x = \Delta\mu / 2$?

In fact, if $q(x)$ had unlimited capacity (for example, if $q$ were itself a two-component mixture), Forward KL would happily set $q^*(x) = p(x)$, dropping to zero in the valley. The reason $q_\phi(x)$ puts mass in the empty middle comes from two forces acting together:

Force 1: The Infinite Penalty for Missing Any Mode

Expand the Forward KL integral:

$$\mathrm{D}_{\mathrm{KL}}(p \parallel q_\phi) = \int p(x) \log \frac{p(x)}{q_\phi(x)}\,dx = \underbrace{\int p(x)\log p(x)\,dx}_{-H(p)\text{ (constant w.r.t. }\phi\text{)}} - \int p(x)\log q_\phi(x)\,dx$$

Look at how the integrand behaves in two different regions of $x$:

Where $p(x) > 0$ and $q_\phi(x) \to 0$: Because $\log q_\phi(x) \to -\infty$, the integrand blows up to $+\infty$. If $q_\phi$ tried to lock onto the left mode and ignore the right mode, data points from the right mode would trigger an infinite penalty. Thus, $q_\phi$ is strictly forbidden from leaving any mode behind.
Where $p(x) = 0$ and $q_\phi(x) > 0$: Because $p(x)$ multiplies the logarithm, the integrand is $0 \cdot \log(0 / q_\phi(x)) = 0$. Forward KL applies zero direct penalty when $q_\phi$ places mass in empty space.

Force 2: The Unimodal Shape Constraint (Collateral Damage)

Because $q_\phi(x) = \mathcal{N}(x; \mu_q, \sigma_q^2)$ is restricted to a single Gaussian bell curve, it is mathematically unimodal. When $\Delta\mu = 8$, a single Gaussian cannot be positive at the left mode ($x = 0$), dip to zero in the middle valley ($x = 4$), and rise again at the right mode ($x = 8$).

Forced to keep $q_\phi(0) > 0$ and $q_\phi(\Delta\mu) > 0$ simultaneously, the only option available to a single bell curve is to center its mean at the midpoint ($\mu_q = \Delta\mu / 2$) and inflate its variance $\sigma_q^2$ so its tails reach both modes. The probability mass sitting in the valley at $x = \Delta\mu / 2$ is pure collateral damage of stretching a unimodal distribution across a bimodal target.

Exact Closed Form: Forward KL = Moment Matching

For any exponential family distribution $q_\phi(x)$, minimizing Forward KL $\mathrm{D}_{\mathrm{KL}}(p \parallel q_\phi)$ has a remarkable analytical solution: moment matching. Differentiating $\mathbb{E}_p[-\log \mathcal{N}(x; \mu_q, \sigma_q^2)]$ with respect to $\mu_q$ and $\sigma_q^2$ and setting to zero gives:

$$\mu_q^* = \mathbb{E}_{x \sim p}[x] = \pi_1 \mu_1 + \pi_2 \mu_2, \qquad \sigma_q^{2*} = \mathrm{Var}_{x \sim p}(x) = \sigma^2 + \pi_1\pi_2(\Delta\mu)^2$$

Notice how $\sigma_q^*$ grows linearly with mode separation $\Delta\mu$: at $\Delta\mu = 10$ with $\pi_1 = 0.5$ and $\sigma = 1$, the fitted standard deviation is $\sqrt{1 + 0.25 \times 100} = \sqrt{26} \approx 5.10$.

3. Reverse KL: Zero-Forcing and the $\log 2$ Mode-Splitting Threshold

Now flip the arguments and minimize $\mathrm{D}_{\mathrm{KL}}(q_\phi \parallel p)$:

$$\mathrm{D}_{\mathrm{KL}}(q_\phi \parallel p) = \int q_\phi(x) \log \frac{q_\phi(x)}{p(x)}\,dx = \underbrace{-\mathbb{E}_{x \sim q_\phi}[\log p(x)]}_{\text{Inaccuracy (avoid low }p\text{)}} \underbrace{- H(q_\phi)}_{\text{Entropy bonus}}$$

Here the expectation is taken under $q_\phi(x)$. This creates a tug-of-war between two terms:

The Zero-Forcing Term ($-\mathbb{E}_{q_\phi}[\log p(x)]$): Wherever $p(x) \approx 0$, $-\log p(x) \to +\infty$. To prevent the expectation from blowing up, $q_\phi(x)$ is forced to be zero everywhere $p(x)$ is zero.
The Entropy Bonus ($-H(q_\phi) = -\frac{1}{2}\log(2\pi e \sigma_q^2)$): Because Gaussian entropy $H(q_\phi) = \frac{1}{2}\log(2\pi e \sigma_q^2)$ enters with a minus sign, increasing $\sigma_q$ lowers the objective. This prevents $q_\phi$ from collapsing into an infinitely narrow spike (Dirac delta) at the highest point of $p(x)$, pushing $q_\phi$ to fill the width of a mode.

Why Mode-Seeking Is Only a Corollary of Zero-Forcing

As Tuan Anh Le observed, people often use zero-forcing and mode-seeking interchangeably, but mode-seeking is merely a byproduct of zero-forcing, and it does not always happen even when $p(x)$ is bimodal!

We can quantify this threshold exactly. Suppose $p(x) = \frac{1}{2}\mathcal{N}(0, \sigma^2) + \frac{1}{2}\mathcal{N}(\Delta\mu, \sigma^2)$ has two well-separated modes, and $q_\phi(x)$ locks onto one of them, say the right mode $q_\phi(x) = \mathcal{N}(\Delta\mu, \sigma^2)$. On that mode, $p(x) \approx \frac{1}{2}q_\phi(x)$, so the Reverse KL divergence approaches:

$$\mathrm{D}_{\mathrm{KL}}(q_\phi \parallel p) \approx \int q_\phi(x) \log \frac{q_\phi(x)}{\frac{1}{2}q_\phi(x)}\,dx = \log 2 \approx 0.6931\text{ nats}$$

(By symmetry, locking onto the left mode at $x = 0$ costs the exact same $\log 2\text{ nats}$; more generally, locking onto a well-separated mode with weight $\pi_k$ costs $-\log \pi_k\text{ nats}$.) This explains all three stages of the Reverse KL optimization landscape as the modes pull apart:

When $2\sigma < \Delta\mu < 4.65\sigma$ (Panel 2): Even though $p(x)$ already has two distinct peaks, the valley between them is shallow enough that only a single centered minimum exists at $\mu_q = \Delta\mu / 2$.
When $4.65\sigma \le \Delta\mu < 5.60\sigma$ (Panel 3, the dotted curve): Two mode-locked local minima emerge near $x = 0$ and $x = \Delta\mu$ (shown by the dotted red curve at $x \approx 4.89$ in Panel 3, costing $\approx 0.67\text{ nats}$), coexisting with the centered mass-bridging minimum (the solid red curve, costing $0.48\text{ nats}$). Because bridging the valley still costs less than the mode-locked fit, the centered curve remains the global optimum.
When $\Delta\mu > 5.60\sigma$ (Panel 4): At $\Delta\mu \approx 5.60\sigma$, the centered bridging cost ($\approx 0.684\text{ nats}$) crosses the mode-locked cost (which approaches $\log 2 \approx 0.693\text{ nats}$ as the modes separate). Past this crossover, the two mode-locked solutions ($x = 0$ and $x = \Delta\mu$) tie as the global minima when $\pi_1 = 0.5$, and a gradient-based optimizer snaps onto whichever peak it was initialized closer to.

Why Is the Mode-Seeking Curve Twice as Tall as $p(x)$?

In Panel 4 (and already emerging in Panel 3, where the dotted local minimum reaches $\approx 0.35$ while slightly widened to $\sigma_q \approx 1.14$), notice that once the modes are well separated and Reverse KL locks onto the right mode at $x = \Delta\mu$, its peak ($\approx 0.40$) stands twice as tall as the peak of $p(x)$ ($\approx 0.20$) right underneath it. Why doesn’t $q_\phi^*(x)$ match the height of $p(x)$?

Because every valid probability density must integrate to $1$ ($\int q_\phi(x)\,dx = 1$):

The target mixture $p(x)$ splits its total probability mass ($1.0$) equally between two modes ($50\%$ around $x = 0$ and $50\%$ around $x = \Delta\mu$). Thus each peak of $p(x)$ has height $\frac{0.5}{\sqrt{2\pi}\sigma} \approx 0.20$.
The mode-seeking Gaussian $q_\phi^*(x)$ abandons the left mode and concentrates $100\%$ of its probability mass ($1.0$) into that single right-hand bell curve of width $\sigma_q \approx \sigma = 1$. Packing twice the probability mass into the same width forces its peak height to double to $\frac{1.0}{\sqrt{2\pi}\sigma} \approx 0.40$, which is also where the unavoidable $\log\frac{1.0}{0.5} = \log 2$ KL divergence comes from!

4. How This Shapes Modern Generative Models & LLMs

The choice between Forward KL and Reverse KL is not just a textbook exercise on 1D Gaussians; it is one of the central design trade-offs in modern foundation models:

Training Paradigm	Effective Divergence	Practical Consequence
Maximum Likelihood (MLE)Autoregressive LLMs, Diffusion pretraining	Forward KL $\mathrm{D}_{\mathrm{KL}}(p_{\text{data}} \parallel q_\theta)$	High recall / diversity, risk of hedging: Because missing any training example incurs a huge $-\log q_\theta(x)$ loss, the model covers every mode of the data distribution. If model capacity or sampling steps are too small, it averages over modes (producing blurry images or overly hedged text).
Variational Inference (ELBO)VAEs, Bayesian posteriors	Reverse KL $\mathrm{D}_{\mathrm{KL}}(q_\phi(z\|x) \parallel p(z\|x))$	Tractable without normalizing constant $p(x)$: Since $p(z\|x) = p(x,z)/p(x)$, taking the log splits the intractable marginal $p(x)$ into an additive $+\log p(x)$ term that is constant w.r.t. $\phi$. Approximate posteriors are well-known to underestimate posterior variance.
Score Distillation & Few-Step GeneratorsSDS, DMD, step-distilled diffusion	Reverse KL $\mathrm{D}_{\mathrm{KL}}(q_\theta \parallel p_{\text{teacher}})$	Sharpness over diversity: When compressing a multi-step diffusion model into a 1 to 4 step generator, minimizing Reverse KL forces generated samples to stay strictly on the teacher’s high-density manifold, yielding crisp images at the cost of reduced sample diversity.
RLHF & Policy OptimizationPPO / GRPO with KL penalty	Reverse KL $\mathrm{D}_{\mathrm{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$	Mode-seeking alignment: Optimizing expected reward minus $\beta\,\mathrm{D}_{\mathrm{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$ corresponds to Reverse KL against the tilted reward distribution $\pi_{\text{ref}}(y\|x)\exp(r(x,y)/\beta)$, concentrating the LLM onto high-reward response modes while zero-forcing implausible tokens.

Forward vs. Reverse KL Divergence

\(\displaystyle \min_\phi \; \mathrm{D}_{\mathrm{KL}}(p \parallel q_\phi)\)

\(\displaystyle \min_\phi \; \mathrm{D}_{\mathrm{KL}}(q_\phi \parallel p)\)