Comparing a distribution with a model
A probabilistic classifier or density model assigns probabilities, not just labels. A model can put too little probability on an event that occurs, and logarithmic loss penalizes that choice. To reason about the loss, we must distinguish the distribution generating the comparison from the model being evaluated, and handle zero probabilities explicitly.
Definition 9.1.1
(Entropy, cross-entropy and relative entropy)
Let and be probability distributions on the same finite nonempty alphabet: their entries are nonnegative and each sums to one. All logarithms are natural. Define
If but for any , define . Terms with contribute zero, including when . Define when every such , and set it to otherwise. These conventions avoid treating an undefined as an ordinary product.
The nonnegativity theorem
Theorem 9.1.2
(Gibbs inequality on a finite alphabet)
For any such , , with equality exactly when . Also , allowing the value on either side. Thus for fixed , cross-entropy is minimized uniquely at over all distributions on the alphabet.
Proof
(The tangent-line bound on the support of p)
If a positive has , then and , so the claim holds. Otherwise wherever . For we have , with equality only at . Apply this to for each with , multiply by and sum:
Hence . Equality forces for every with ; these already sum to one, so at the remaining coordinates. Conversely gives .
For the identity, when on the support we have for each with , and summing gives . If some positive has , both and equal . For fixed the entropy is a finite constant, so minimizing over means minimizing , whose unique minimizer is .
Proof X-ray
What each hypothesis does in the Gibbs inequality
The one non-obvious move is the tangent-line bound at . It replaces a logarithm of ratios by a linear expression, so the sum collapses to and nothing else about and is needed. Strictness of the bound for is what turns into equality only when . The proof treats the support of separately for two reasons. Terms with are absent from , so can be less than one; that loss is exactly the slack , and it is why equality forces to vanish off the support. A zero on the support makes undefined, so that case is handled by the value . Both sums equal to one are used at the final step: for an unnormalized the conclusion can fail, since and on a one-letter alphabet would give .
Worked examples
Worked example 9.1.3
(A mismatched fair model)
Take and . The model cross-entropy is , while
The difference is nonnegative and vanishes only when the model equals . Reversing the arguments gives : relative entropy is generally asymmetric. If instead , then because the second outcome has positive -probability but zero model probability.
Counterexample mode
A zero that matters
For and , the second term contributes zero even though ; the divergence is . Reversing the distributions assigns positive mass under the first argument to an outcome assigned zero by the second, so . A zero in the model and a zero in the target have different roles.
Log loss on observed data
Proposition 9.1.4
(Average log loss is empirical cross-entropy)
For observations in the alphabet, let count outcome and . For every distribution on the alphabet,
and both sides are infinite when an observed outcome has . Consequently the average log loss is minimized uniquely at .
Proof
(Grouping the observations by outcome)
Among the terms exactly equal , so . Outcomes with contribute nothing, which matches the convention in . If an observed outcome has , both sides are . The statement about the minimizer is the Gibbs inequality with .
This identity is algebraic for the given data; it does not require the observations to be independent. Under an independent-sampling model, minimizing this quantity is also maximum likelihood. The nonnegativity theorem says that the unconstrained minimizer over all distributions is ; within a restricted model family the best achievable model may differ.
Worked example 9.1.5
(Training loss of three models on one data set)
Let the data contain outcome one three times and outcome two once, so and . For the uniform model the average loss is , an excess of over , as in the first example. For the overconfident model ,
an excess of about . So the overconfident model has lower training loss than the uniform one, and both exceed the minimum attained at . For the average loss is infinite, because outcome two was observed.
Worked example 9.1.6
(A zero in the training frequencies)
Let the alphabet have three outcomes and the training counts be , so . The unique minimizer of the training loss is , with loss . Its third coordinate is zero, so a later observation of outcome three has log loss . The smoothed distribution , which adds one pseudo-count to every outcome, sums to one and gives the finite loss on outcome three. Its training loss is , larger than , as the proposition requires. Here training loss and held-out loss pull in different directions, and the training data alone cannot decide which is preferable.
Training loss alone does not measure future performance or establish calibration. Held-out evaluation and a justified sampling model are separate questions. The entropy of a target is a property of that target; changing logarithm base changes units and rescales every finite term.
Common mistake
(The first argument is the data distribution)
In the first argument supplies the weights. For log loss on data it is , and the second argument is the model. With and we found but , so quoting one for the other, or calling a distance between two distributions, is wrong. The two orders also differ in which zero is dangerous, as the counterexample above shows.
Exercises
Quick check
(Compute for .)
Look at the ratios .
Solution · Answer
It is zero, because both ratios on the support equal one, so every logarithm vanishes. This is the equality case of the Gibbs inequality.
Quick check
(For and , compute and determine .)
Only outcomes with positive probability under the first argument contribute.
Solution · Answer
, since the second term is omitted because . For the second outcome has while , so .
Quick check
(A data set has three occurrences of outcome one and one of outcome two. What is its empirical distribution? What distribution minimizes empirical cross-entropy over the full simplex?)
Apply the proposition on average log loss.
Solution · Answer
The empirical distribution is . It uniquely minimizes the empirical cross-entropy over all distributions, with minimum value .
Quick check
(Why is a model assigning zero to an observed outcome problematic for log loss?)
Write the loss of that observation.
Solution · Answer
The corresponding negative logarithm is infinite, so the average loss is however well the model fits the other observations. Equivalently, the product likelihood is zero and its negative log is not finite.