Why a probability needs its own assessment
Two classifiers can make exactly the same hard decisions while reporting very different probabilities. Under a threshold of one half, forecasts of and both produce class one. If the event happens only about half the time in the relevant group, those probability statements deserve different assessments even though their zero-one decisions agree. Conversely, improving probabilities can leave all decisions unchanged for one loss but change them for another.
A forecast is therefore more than a label with a confidence decoration. It is a numerical claim about an event, assessed against a specified population. This note develops that claim for binary labels and finitely many score values. The finite setting lets us prove every identity, display every probability and calculate a complete cost-sensitive example. The table is deliberately synthetic; its exact fractions are a teaching construction, not findings from a real dataset.
The population and the forecast
Definition 28.1.1
(Finite scores, conditional frequencies and calibration)
Let have a specified joint probability distribution, with and a forecast taking distinct values in . For each positive-mass score value define
The forecast is calibrated for this population if for every . Its binary Brier loss is , and its population Brier score is ; smaller values are better.
The probabilities can describe a finite population sampled uniformly or a more general distribution. No independence assumption is needed for identities concerning this single joint law. Independence will matter when a separate sample is used to estimate its probabilities.
Calibration is relative to the information retained in the score and the population being assessed. It does not assert that every individual assigned has a hidden true probability of given all available features. It asserts that the conditional event frequency given this score is . A score calibrated in one population need not remain calibrated after its joint distribution with the label changes.
Definition 28.1.2
(A strictly proper probability loss)
A binary probability loss is strictly proper if, for every , the expected loss for is uniquely minimized by reporting . This concerns expected loss under a probability law, rather than the loss of a single realized label.
For the binary Brier loss, direct expansion gives
The last square is nonnegative and vanishes exactly at , including the endpoints. This proves strict propriety. A single positive outcome instead has its smallest realized loss at ; that observation does not justify predicting one before every uncertain event. Expected loss and hindsight loss answer different questions.
Some authors sum squared errors for both coordinates of the probability vector . For a binary label that convention gives . Every Brier quantity here uses the single-coordinate convention above.
Reliability, resolution and uncertainty
Theorem 28.1.3
(An exact finite Brier decomposition and an oracle recalibration)
For the stated finite distribution, define
Then
The population recalibration , defined by when , is calibrated, even if several coincide, and
Moreover, uniquely minimizes Brier risk among numerical forecasts that assign one fixed value to each original score group.
Proof
(Condition within each score group before adding groups)
Conditional on , strict propriety gives
Average with weights . Since ,
These two equalities prove the decomposition. Replacing by removes exactly the squared discrepancies. More generally, replacing it by any yields the same irreducible term plus , proving uniqueness group by group.
To prove calibration after merging, fix a value taken by . Its preimage consists exactly of score groups with . Thus
This conditional probability is defined because the denominator is positive. No one-to-one assumption on the recalibration map was required.
Reliability here names a nonnegative discrepancy: smaller REL is better, and zero is equivalent to calibration on the score's support. Resolution measures how much the conditional event frequencies vary around the overall prevalence. For fixed reliability in the same population, greater resolution lowers Brier loss. Uncertainty is the Brier loss of always forecasting in this same population. Comparing raw scores across populations with different prevalence can therefore mix forecasting quality with a different baseline task.
The map in the theorem is an oracle: it uses the population joint law. A fitted recalibration map estimates it and need not improve risk on fresh observations. The exact improvement identity does not authorize substituting noisy estimates and declaring the same improvement.
Proof X-ray
Two different sources of squared error
The first conditional identity separates disagreement between a forecast and its group frequency from random labels within that group. The second identity separates variation between group frequencies from overall label uncertainty. Both use the same score groups and the same probability weights. A histogram with arbitrarily widened bins changes those groups; it needs an additional calculation rather than inheriting this exact decomposition for the original scores.
A complete finite table
Worked example 28.1.4
(A synthetic population with sixty equally likely records)
Construct sixty distinguishable records and choose one uniformly. In group A, thirty records receive score , with three positive labels. In group B, twenty receive , with ten positive labels. In group C, ten receive , with eight positive labels. The complete distribution is therefore:
| Group | Score s | Number | Positive | Negative | Weight p | Event frequency q |
|---|---|---|---|---|---|---|
| A | 1/5 | 30 | 3 | 27 | 1/2 | 1/10 |
| B | 3/5 | 20 | 10 | 10 | 1/3 | 1/2 |
| C | 9/10 | 10 | 8 | 2 | 1/6 | 4/5 |
The population prevalence is . Every score exceeds its group frequency by , so . The other terms are
We can check the result without using the theorem. Total squared losses within A, B and C are respectively , and . Dividing their sum by sixty gives . Oracle recalibration changes the forecasts to , reducing Brier score to . The reduction is exactly .
If instead these counts describe one constructed sample from an unspecified population, the same arithmetic is an exact empirical identity only. The table alone would not prove that its group frequencies are population probabilities.
Common mistake
(A perfectly calibrated forecast is not necessarily a good one)
Reliability is only one of three terms. In the sixty-record table, always reporting the prevalence is perfectly calibrated: its single group has event frequency , so and . Its Brier score is therefore . The three-group forecast above has yet Brier score , which is better. A small REL says that the reported numbers are honest, not that they distinguish cases.
From probabilities to decisions with costs
Definition 28.1.5
(Two error costs and a score-based action)
Let a false positive cost , a false negative cost , and correct decisions cost zero. For any binary action rule , let be its expected cost under the same population. Write
The displayed rules choose action one at a tie. The population conditional frequency, rather than the raw score, defines the oracle action .
Theorem 28.1.6
(Score-based Bayes risk and a calibration-to-cost bound)
Among all rules using only , minimizes expected cost. Its excess-risk comparison with the raw-score threshold is
The exact identity includes ties because their conditional regret is zero. The bound compares two rules with access to the same score information; it is not a guarantee against a rule using every original feature.
Proof
(Compare the two conditional costs, then locate a crossed threshold)
In group , action one has conditional cost and action zero has cost . Their difference, one minus zero, is . Thus choosing one is optimal when , and choosing zero is optimal when . A randomized choice within the group has a convex combination of these two costs, so cannot improve on their minimum.
If the two threshold rules disagree, choosing the nonoptimal action adds ; if they agree, regret is zero. Averaging proves the identity. On disagreement, and are on opposite sides of the threshold, allowing an endpoint at a tie. Hence . Finally,
by weighted Cauchy–Schwarz. This proves the bound without assuming that different observations are independent.
Worked example 28.1.7
(The same table, with a specified asymmetric loss)
Take and . The decision threshold is . Raw forecasts select actions in A, B and C. The oracle frequencies select : group B's event frequency is , below .
For the sixty equally likely records, raw-threshold total cost is , giving risk . After oracle recalibration, the total is , giving . The improvement is . The regret formula gives the same result:
The bound is , a valid but loose upper bound. With equal error costs and the declared tie rule, both forecasts instead select . Recalibration still improves Brier score, but does not change that decision risk. Probability assessment and decision assessment are related through the chosen loss; they are not interchangeable.
Common mistake
(Using the threshold one half when the two errors cost different amounts)
With and , group B in the table has event frequency . Acting at the default threshold chooses action one there, at expected cost per record, whereas action zero costs . The correct threshold is . Averaged over the population, the default choice costs more than the best score-based rule. The threshold comes from the costs, not from the probability scale.
Counterexample mode
Perfect calibration can conceal a completely informative feature
Let be a fair binary variable and always report . This score is calibrated: its only group's event frequency is . Under equal unit error costs, its thresholded action always selects one, with risk . The feature-based rule has risk zero. There is no contradiction with the theorem: that theorem compares rules restricted to , which has discarded .
More generally, when is a finite-valued deterministic function of a feature defined jointly with , write . Conditional expectation gives . Expanding the square and conditioning the cross term on yields
The second term measures probability information lost by retaining only . It is in this example even though REL is zero. Calibration repairs the score's probability scale; it cannot restore a feature that the score no longer records.
What binning and finite evaluation do
Worked example 28.1.8
(Why a broad calibration bin changes the identity)
For any finite partition variable whose cells all have positive probability, write and . Expanding conditional squares gives
The first two terms are the conditional Brier loss of the coarsened forecast . The last two terms describe what replacing by that group mean changes. They vanish for bins containing one exact score value, but need not vanish for a broad bin.
For an exact counterexample, let with two equally likely possibilities, zero and one, and put both into one bin. The original Brier score is zero. The bin mean and bin event frequency both equal , so the bin-level formula without the corrections gives . Here conditional variance and covariance both equal , and their correction is , restoring the true answer. A visually perfect one-bin reliability diagram has not measured the original forecast's resolution.
To assess a learned calibration map, first separate its fitting observations from the final evaluation observations. Fit the base predictor and every selected transformation without the final test labels. Estimate the calibration map using a permitted calibration portion; if its family or tuning parameter is chosen, keep that selection within development data as well. Freeze both the map and any decision threshold before inspecting the final test labels. Specify the error costs and target population rather than selecting whichever cost makes the test result look favorable.
For independent identically distributed test pairs from that target population, conditional on the already fitted pipeline, the mean test Brier loss has expectation equal to its population Brier risk. The same argument works for the prespecified bounded cost loss: condition on the fitted rule and average test losses. This argument uses independence between the test sample and all fitted or selected choices. It does not say that a test estimate equals its expectation, or that an empirically fitted calibration curve is an unbiased population curve.
Even perfectly calibrated forecasts show sampling noise. If one fixed score group has population frequency and independent labels, then has mean and variance . Therefore
For and , this expected empirical reliability discrepancy is , although the population discrepancy is zero. The formula assumes a fixed group size and independent Bernoulli labels with the stated probability. Data-dependent bins, repeatedly selected diagrams and fitted-score reuse need additional analysis.
Concept lensStructural
Three questions for one probability forecast
Brier risk asks how well the numerical forecasts predict binary labels. Calibration asks whether each retained score group has the reported event frequency. Decision risk asks what happens after a particular cost threshold turns those numbers into actions. The same finite table answers all three, but its answers contain different information. An evaluation report should identify the question before presenting its number.
Practice
Quick check
(A Bernoulli event has probability . Compare expected binary Brier losses for forecasts and . Explain why one realized negative label does not refute strict propriety.)
Expand the two conditional expected losses before substituting any observed outcome.
Solution · Answer
For , the loss is always . For , expected loss is . Their difference is . A negative label has probability under the model and gives larger realized loss to than to . Strict propriety concerns the expectation over both possible labels, not domination for each individual outcome.
Quick check
(Two score groups have equal weights, scores and , and event frequencies and . Calculate prevalence, REL, RES, UNC, raw Brier score and oracle-recalibrated Brier score.)
Keep the squared discrepancy separate from variation between group frequencies.
Solution · Guided solution
The prevalence is . Each absolute score discrepancy is , giving . Each frequency differs from prevalence by , giving , while . Thus raw Brier score is . Oracle recalibration removes , leaving . Both original and recalibrated forecasts choose the same actions at threshold .
Quick check
(In the sixty-record table, set false-positive cost to four and false-negative cost to one. Find raw and oracle actions, their common risk and the tie in the oracle rule.)
At a conditional probability of , calculate the cost of each action directly.
Solution · Guided solution
The threshold is . Raw forecasts select . Oracle frequencies select the same actions under the declared choose-one-at-equality rule. In C, choosing one costs , while choosing zero costs as well, so either oracle action is optimal. The sixty-record cost is , giving common risk . Choosing zero in C would instead incur eight false negatives there, again costing eight. Recalibration improves Brier score without improving this cost.
Quick check
(A finite domain has distinct inputs, one independent fair training label per input, and distinct prespecified raw scores. An empirical recalibrator replaces each score by its single observed label. Evaluate its training Brier score and its expected Brier score for an independent fair label at a uniformly chosen input. Compare with always reporting .)
Condition on the fitted lookup table before evaluating a fresh label.
Solution · Answer
Each fitted score equals its training label, so training Brier score is zero. Conditional on any fitted lookup table, a fresh fair label matches its stored zero-or-one forecast with probability one half. Expected fresh loss is therefore at every input and after averaging inputs. Always reporting has loss for either fresh label. The apparent perfect empirical calibration was fitted using those same labels; it is not evidence of population calibration or generalization. This construction supplies no iid-input training claim: it explicitly uses one label at each distinct finite-domain input.