A classifier score ranks examples; a threshold turns that ranking into decisions. The same scores therefore produce many confusion matrices, not one. A receiver operating characteristic (ROC) curve shows how the true positive rate changes with the false positive rate as the threshold moves. Its area (AUC) summarizes ranking, but it does not by itself tell us precision or the best threshold.
Roadmap and assumptions
First draw the confusion matrix at a fixed threshold. Next sort distinct scores, moving all tied examples together. Then compute AUC in two ways: trapezoids between ROC points, and positive–negative pair comparisons. Finally use prevalence to interpret precision and decision costs. Assume larger scores mean stronger evidence for the positive class; classify scores at least a threshold as positive. Both classes must occur for empirical ROC and AUC to be defined.
Definition 7.2.1
(ROC point and AUC)
Let and be the numbers of positive and negative examples. The true positive rate is ; the false positive rate is . A ROC point is . At a threshold above every score the point is ; at or below the minimum score it is . The ROC curve joins the points obtained from successive distinct score levels by line segments, and the AUC is the area under that piecewise-linear curve.
Why tied scores move together
Suppose one score level contains positives and negatives. Lowering the threshold through that level changes TPR by and FPR by simultaneously. A deterministic score-only rule cannot accept one member of the tied group and reject another. A point strictly inside the line segment joining the two threshold points can instead be achieved in expectation by randomly choosing the more inclusive rule with the appropriate probability, independently of the true class. A separate feature or a fixed tie-breaking rule may also split the group, but it is a different decision rule and can yield different rates.
Worked example: two routes to the same AUC
Worked example 7.2.2
(Two routes to the same AUC)
Take positive scores and negative scores . The table starts above the maximum and then includes each entire score group. Counts are cumulative.
| Included score level | TP | FP | ROC point |
|---|---|---|---|
| none | 0 | 0 | |
| 1 | 0 | ||
| 1 | 1 | ||
| 2 | 2 | ||
| 3 | 2 | ||
| 3 | 3 |
For linearly interpolated ROC points, the horizontal, diagonal and final horizontal contributions are , and . Vertical segments have zero area. Thus .
Alternatively compare each positive with each negative. Award one for a higher positive score, half for a tie, and zero for a lower positive. The three positive scores contribute , , and , respectively. There are pairs, giving again.
Theorem 7.2.3
(AUC as a pairwise comparison)
Suppose and group the examples by distinct score level. Then the area under the piecewise-linear ROC curve equals
where ranges over pairs of one positive and one negative example.
Proof
(Summing trapezoids level by level)
Order the distinct score levels from highest to lowest as , and let level contain positives and negatives. Before level is included, the true positive rate is ; afterwards it is . Including level moves the point right by and up by , so the trapezoid under that segment has area
When the segment is vertical and the area is zero, consistent with the formula. Summing over gives . The first sum counts the pairs whose positive lies in a strictly higher level than its negative, that is, ; the second counts the pairs in the same level, . This is the displayed expression.
The pairwise expression describes a randomly chosen positive and negative from the empirical data, with half credit for a tie. It is not the probability that an arbitrary single prediction is correct.
Proof X-ray
Why a tie earns exactly one half
The proof has one idea: the horizontal width of a segment is proportional to the number of negatives in the level, and its average height is proportional to the number of positives strictly above that level plus half of the positives in it. The half is the average of the heights before and after the level, because ties move the point along a straight chord from to rather than in two separate steps.
Grouping is where the hypothesis is used. If tied examples were instead ordered arbitrarily and added one at a time, the curve would be a staircase whose area depends on the order chosen, and the pairwise count would give full or zero credit to tied pairs. The half credit is therefore a property of the chord, not an independent convention. The identity also needs for the division, and no assumption about the score distribution.
The role of the chord shows up in the smallest possible case.
Counterexample mode
Ties are not strict wins
Claim: AUC is the fraction of positive–negative pairs with a strictly higher positive score. Take one positive and one negative example, both with score . There is one pair and no strict win, so the claim gives . But there is a single level, containing both examples, so the ROC curve goes from straight to and the area is . The corrected statement gives . Ordering the tied pair arbitrarily instead would give a staircase of area or , which describes a different, order-dependent rule.
Worked example 7.2.4
(All scores equal)
Let there be positives and negatives, all with the same score. The only ROC points are and , so . By pairs, all pairs are ties, giving . A constant score carries no ranking information, and AUC reports exactly that.
Worked example 7.2.5
(A cost-based threshold on calibrated posteriors)
Suppose and and the scores are calibrated posteriors. The rule predicts positive when . For , predicting positive has expected cost and predicting negative has , so predicting positive is cheaper. For the costs are and , so predict negative. At both costs equal , so either decision is permissible. The threshold is far below because a missed positive costs four times as much as a false alarm.
ROC is not precision
Let prevalence be , sensitivity , and false positive rate . Whenever , Bayes' rule gives
For and , precision is at prevalence , but at prevalence . These comparisons hold the class-conditional rates fixed; a changed population can also change those rates. If the denominator is zero, precision is undefined, not automatically zero or one. A high AUC alone does not specify precision, calibration, or the cost-optimal threshold.
Scores and posterior probabilities
A score need not be a calibrated probability. If a score is a calibrated posterior , false positives cost , false negatives cost , and correct decisions have zero cost, the conditional expected costs are for predicting positive and for predicting negative. Predict positive when
with either decision permissible at equality. This assumes the stated costs and correct posterior for the target population. A ranking score or an AUC alone cannot justify applying this probability threshold.
Common mistake
(Reading AUC as the chance a prediction is correct)
An AUC of does not mean of predictions are right. It compares scores across positive–negative pairs; the accuracy, precision and best threshold are separate questions that depend on prevalence and costs.
Common mistake
(Counting a tied pair as a loss or a win)
When a positive and a negative share a score, giving full credit or no credit changes the AUC. In the main worked example the positive and the negative with score should contribute ; counting them as a win would give instead of in that example, and counting them as a loss would give .
Counterexample mode
Equal AUC does not mean equal curves
Claim: two scorers with the same AUC are interchangeable. Take positives and negatives. Scorer A gives positives and negatives ; scorer B gives positives and negatives . By pairs, A has wins and ties, so . B has ties and wins, again . Yet A's points are and B's are : A reaches with no false positives, while B reaches no positive without . The repair is to compare the curves, or the range of that the application can tolerate.
Practice questions
Quick check
(Positive scores are and negative scores are . Compute the AUC by pairs.)
List all four pairs; ties earn half credit.
Solution · Answer
The pairs are win, win, loss and tie. The total is over pairs, so .
Quick check
(A classifier has and . What is its precision at prevalence ?)
Use Bayes' rule with the prevalence.
Solution · Answer
Precision is .
Quick check
(With and , at what calibrated posterior is the decision a tie?)
Set the two expected costs equal.
Solution · Answer
The costs and are equal when , so . This agrees with .
Checklist
- State the score direction and threshold convention; include endpoints and move ties together.
- Divide TP by positives and FP by negatives; check that both denominators are positive.
- Distinguish deterministic points from randomized interpolation and use half credit for ties in trapezoidal AUC.
- Compute precision from prevalence and class-conditional rates, checking its denominator.
- Use posterior decision thresholds only under calibration and explicit cost assumptions.