A linear classifier produces a number . A logistic regression produces a number between and . Neither number is a decision. A decision appears only when we choose a threshold and declare every case at or above it to be positive. Evaluating a classifier therefore means evaluating two things: the ordering the score induces on cases, and the threshold applied to that ordering. This note builds the vocabulary for both, works through every calculation on a small dataset, and explains what the resulting numbers do and do not guarantee.
Scores, thresholds and decisions
Each case has an actual label , where means "positive". A fitted model assigns a real-valued score , where larger values mean "more positive". For a threshold , the decision rule is
The inequality is non-strict. Tie convention: a score exactly equal to the threshold is classified as positive. This convention is fixed throughout. Changing it to changes the counts at any threshold that coincides with an observed score, so a reported metric is incomplete unless the convention is known.
For logistic regression, the probability score and the linear score give the same family of decisions. The function is strictly increasing, so for ,
A threshold of on the probability is a threshold of on the linear score. The logistic output may look like a probability, but that alone does not show that it is one. We return to this distinction under asymmetric costs below.
The confusion table
Fix a threshold and a finite evaluation sample of cases. Every case falls into exactly one cell. Rows are actual classes and columns are predicted classes.
| Predicted positive () | Predicted negative () | Row total | |
|---|---|---|---|
| Actual positive () | TP | FN | |
| Actual negative () | FP | TN | |
| Column total |
Each standard metric is a ratio of counts. What distinguishes one metric from another is its denominator, because the denominator determines which question the metric answers.
Definition 5.7.1
(Confusion-table metrics)
- Accuracy : what fraction of all cases is classified correctly?
- Recall, also called sensitivity or true positive rate (TPR), : of the actual positives, what fraction is flagged?
- Specificity, or true negative rate (TNR), : of the actual negatives, what fraction is cleared?
- False positive rate (FPR) : of the actual negatives, what fraction is wrongly flagged?
- Precision, or positive predictive value (PPV), : of the flagged cases, what fraction is actually positive?
Recall and specificity condition on the actual class, which is a row of the table. Precision conditions on the prediction, which is a column. This difference will matter when prevalence changes.
Common mistake
(Reporting an undefined metric as a number)
If , recall is . If nothing is predicted positive, precision is . These quantities are undefined. Some software returns , returns , or issues a warning. Any of these conventions is acceptable only if it is stated, and none of them should be mistaken for a measurement.
Theorem 5.7.2
(Count form of the F1 score)
Suppose . Then precision and recall are both defined and positive, and their harmonic mean is
The right-hand side is defined whenever . When and , the count form equals , even if precision itself is undefined. When , the denominator is zero and is undefined.
Proof
(Deriving the count form)
Write and . Then and , so
Dividing the first expression by the second and canceling gives . Finally, .
The cancellation step uses . That is why the cases with are not covered by the harmonic-mean formula and have to be treated as an extension of the count formula.
Notice also that TN never appears in the count form. F1 measures how well the positive class is handled. It is not a version of accuracy.
Counterexample mode
High accuracy with zero recall
Suppose of cases are positive, so and . A "classifier" that predicts negative for every case has
Its accuracy is . Its recall is , so it detects nothing. Its precision is , which is undefined. Its count-form F1 is .
At low prevalence, accuracy is dominated by the negative class. A high accuracy value can hide complete failure on the positive class.
A worked dataset with a tie
Worked example 5.7.3
(Six scored cases)
The evaluation sample has six cases. The three positives have scores . The three negatives have scores . Sorted from the highest score down, the cases are:
Threshold . The cases flagged positive are those with , namely the scores , and .
- Two of these are positive, so .
- One is negative, so .
- The positive with score is missed, so .
- The negatives with scores and are cleared, so .
The metrics are:
- precision
- recall
- specificity
- accuracy
The five values coincide here only because of the symmetric counts. They measure different things.
Threshold . Both cases with score now satisfy . The counts are , , , . The metrics are:
- precision
- recall
- specificity
- accuracy
Accuracy is unchanged from , while recall, precision and F1 have all moved.
Which thresholds need to be checked? The decision for each case depends only on whether its score is at least . Between two consecutive distinct observed scores, no decision changes. It is therefore enough to check each distinct score, plus one threshold strictly above the maximum score.
| Threshold | TP | FP | FN | TN | TPR | FPR | Precision |
|---|---|---|---|---|---|---|---|
| 0 | 0 | 3 | 3 | undefined | |||
| 1 | 0 | 2 | 3 | ||||
| 1 | 1 | 2 | 2 | ||||
| 2 | 1 | 1 | 2 | ||||
| 3 | 2 | 0 | 1 | ||||
| 3 | 3 | 0 | 0 |
The two cases with score form a tie block. Any flags both of them, and any flags neither. No deterministic scalar threshold can flag the positive at while clearing the negative at .
The ROC curve
The receiver operating characteristic (ROC) plots one point for each threshold. The horizontal axis is FPR and the vertical axis is TPR. From the table, the achievable points are
The endpoints need care because of the tie convention.
- The point requires a threshold strictly above the maximum score. Using still flags the top case, so it gives , not .
- The point requires a threshold at or below the minimum score, that is .
Lowering the threshold past an untied positive moves the curve straight up by . Lowering it past an untied negative moves the curve straight right by .
Lowering it past the tie block moves both coordinates at once. The curve goes diagonally from to .
The points strictly inside this diagonal segment are not produced by any deterministic threshold. They can be reached in expectation by randomizing. At , flag each tied case independently with probability . Then the expected rates are
As runs from to , the expected point traces the segment. The connected polygon is therefore a convenient summary of the ranking. It is not a list of deterministic operating points.
Empirical AUC
Assume the sample contains at least one positive and at least one negative, so and . The empirical AUC averages over all positive–negative pairs:
In words, it is the fraction of pairs in which the positive outranks the negative, with tied pairs given half credit.
Worked example 5.7.4
(Empirical AUC of the six cases, by pairs and by area)
For the six-case dataset, take each positive in turn and compare it with the three negatives:
- The positive at beats all three negatives: .
- The positive at loses to and beats and : .
- The positive at loses to , ties with and beats : .
Hence
The same number appears as an area. Split the ROC polygon into strips:
- the horizontal step at height has area ;
- the diagonal segment has area ;
- the final step at height has area .
The total is .
Theorem 5.7.5
(Pairwise AUC equals the area under the interpolated ROC polygon)
Join the deterministic ROC points by straight segments, including a diagonal segment across each tie block. The area under the resulting polygon equals as defined by the pairwise formula with half credit for ties.
Proof
(Counting pairs strip by strip)
Lower the threshold one block of equal scores at a time. Suppose a block contains positives and negatives, and that positives have strictly higher scores than the block.
Before the block is crossed, the curve is at height . Afterwards it is at height . The horizontal width of the step is . The trapezoid under this step therefore has area
Now interpret the bracket in terms of pairs. The term counts the pairs in which one of the higher positives strictly beats one of the negatives in this block. The term is the half credit for the tied pairs inside the block.
Every positive–negative pair is counted in exactly one strip, namely the strip belonging to the negative's block. A pair in which the negative outranks the positive contributes in every strip. Summing over all blocks gives the pairwise formula.
Proof X-ray
What the strip decomposition relies on
The one non-obvious move is to attribute every positive–negative pair to the block that contains the negative. Lowering the threshold block by block moves the ROC path monotonically up and to the right, so the strips do not overlap and their areas add. Inside a block, the diagonal segment produces the factor : the area under a straight segment is the average of its end heights, which is exactly half credit for the tied pairs. A staircase through the tie block, up first or right first, would give credit or instead, the two random tie-breaking values. The hypotheses make the normalization by legitimate.
Common mistake
(Reading half credit as partial success)
Suppose the tied pair at were broken at random. If the positive were placed first, the AUC would be . If the negative were placed first, it would be . Each order has probability , so the expected value is . The half credit is thus the expected value of a coin flip. A tie is a failure to discriminate, not a success. A perfect AUC of requires every positive to score strictly above every negative.
AUC measures ranking, not calibration. Suppose is strictly increasing. Then holds if and only if , and holds if and only if . Every pair term is unchanged, so the AUC is unchanged. In particular, and have identical AUC.
Concept lensCombinatorial
AUC as a rank-sum statistic
Since only the order of the scores matters, the AUC can be computed from ranks. Rank all scores from (lowest) to , give each tie block the average of the ranks it occupies, and let be the sum of the positives' ranks. Then
A positive's rank counts itself, every case below it and half of its tied rivals; subtracting removes the positive–positive comparisons. In the six cases the positive ranks are , so . The numerator is the Mann–Whitney statistic.
Calibration, by contrast, concerns the actual values of the scores. Positive scores of and negative scores of produce the same AUC as positive scores of and negative scores of . These two score sets make very different probability claims.
AUC also does not depend on the threshold at all. Choosing a cost-optimal threshold does not change AUC, and AUC is not the quantity a cost-based threshold optimizes.
Precision–recall points
Plotting (recall, precision) for the same thresholds gives the points
At , recall is and precision is undefined, so that threshold contributes no point.
Two cautions apply. First, precision has the predicted-positive count in its denominator. That count mixes the two classes in proportions set by prevalence, so precision–recall curves can shift when prevalence shifts, while ROC coordinates conditioned on each class need not. Second, joining precision–recall points by straight lines does not, in general, describe achievable classifiers. If a curve is drawn, the interpolation convention should be stated rather than presumed meaningful.
Prevalence and positive predictive value
Now move from a sample to a population. Let
Theorem 5.7.6
(Precision as a function of prevalence)
If and , then
Proof
(Bayes' rule on the decision)
By the definition of conditional probability,
The numerator is . For the denominator, apply the law of total probability over the two classes:
The hypothesis is exactly what is needed to make the division legitimate.
Worked example 5.7.7
(One detector at two prevalences)
Take and .
- At : . Fewer than one flag in thirteen is a true positive.
- At : .
The same detector gives very different precision in the two settings.
This calculation assumes that TPR and FPR are the same in both populations. Under dataset shift, that assumption can fail. Instruments change, subpopulations differ and case definitions drift, and any of these can alter the conditional rates themselves.
Moreover, counts from an evaluation sample only estimate population rates. A recall computed from a handful of positives carries substantial sampling uncertainty. Such an estimate is not a population guarantee.
Asymmetric costs and the decision threshold
Suppose we know the true conditional posterior . Suppose also that a false positive costs , a false negative costs , and correct decisions cost nothing.
For a case with features , the conditional expected costs of the two actions are:
- predict positive: ;
- predict negative: .
Predicting positive is at least as good as predicting negative exactly when
At equality both actions have the same expected cost, so either is optimal. Choosing positive at equality matches our convention. Equal costs recover the familiar threshold .
This derivation rests on two strong assumptions: the costs are fixed and known, and the score is the true posterior.
A score that is merely calibrated, meaning , supports a weaker conclusion. Thresholding at the same cost ratio is optimal only among rules that see alone, not among all rules based on . Calibration checked only by bin averages is weaker still. A bin can be right on average while being wrong for the individual scores inside it.
Honest validation
Every choice that looks at data is part of fitting. This includes standardization constants, imputation values, feature selection, regularization strength, and the threshold itself. A defensible protocol has three stages:
- Training data. Fit the preprocessing and the model parameters using the training data only.
- Validation data. Use held-out validation data, or cross-validation within the training data, to compare models and choose .
- Test data. Apply the frozen pipeline once to an independent, untouched test set and report the resulting metrics.
If the test set is consulted repeatedly to tune choices, it has become a validation set. Repeated tuning can induce optimistic selection bias, and the simple holdout interpretation no longer holds.
Even a clean test result has limits.
- Dependence between cases can make the effective sample smaller than it appears. Repeated records from the same individual, or temporal autocorrelation, are common sources of dependence.
- Small class counts produce wide uncertainty in recall and precision.
- Test prevalence may differ from deployment prevalence.
- The test set may not represent the deployment population.
- Labels may be measured with error.
None of these metrics, by itself, supports a causal interpretation or a claim that the classifier is fair across groups.
Exercises
Quick check
(A subsample of cases contains only negatives, and all are predicted negative. Compute every metric that is defined.)
Check every denominator before dividing.
Solution · Guided solution
The counts are and . Accuracy and specificity both equal , and FPR is . Recall, precision and are all and therefore undefined.
Reporting these as or as would describe a convention, not the classifier. With no positives present, the subsample carries no evidence at all about detecting positives.
Quick check
(Take , , , . Compute all the metrics and compare the accuracy with that of the all-negative rule.)
Find the row and column totals first.
Solution · Guided solution
First the totals: , , , and cases are predicted positive. The metrics are:
- accuracy
- recall
- specificity
- FPR
- precision
As a check, the harmonic mean gives
Predicting negative for every case would achieve accuracy . The gain in accuracy is therefore only three percentage points, while recall rises from to .
Quick check
((a) There are two positives and three negatives, and every score equals . Find the AUC. (b) The positives have scores and the negatives have scores . Find the AUC, then find it again after the transformations and .)
Count each positive–negative pair once.
Solution · Guided solution
(a) All six pairs are tied, so . The only deterministic ROC points are and . The score provides no discrimination at all.
(b) There are pairs.
- The positive at beats both negatives: .
- Each positive at ties the negative at and beats the negative at : each.
The total is , so .
The map is strictly increasing. It gives scores against , so every pair outcome is unchanged and the AUC remains .
The map is non-decreasing but not strictly increasing. It merges into , giving positives against negatives . Each positive now earns , so the AUC becomes . Merging scores creates new ties and lowers the AUC.
Quick check
(A detector has and . (a) Compute the PPV at and at . (b) With and , find the optimal posterior threshold. (c) A model's posterior is exact in a population with and gives for some case. Deployment moves to . Assume is discrete with positive class-conditional probabilities at , and the class-conditional distributions of are unchanged (label shift). Find the new posterior and the optimal decision. The same odds calculation applies to suitable class-conditional densities on their common positive support, almost everywhere.)
Work with odds when the prior changes.
Solution · Guided solution
(a) At :
At :
(b) The threshold is .
(c) By Bayes' rule, posterior odds equal prior odds times the likelihood ratio .
In the original population, the posterior odds are and the prior odds are . Hence .
Under label shift, is unchanged. The new prior odds are , so the new posterior odds are
The new posterior is therefore
Since , the optimal action is to predict negative. In the original population, the same case would have been flagged, because . The conclusion depends entirely on the label-shift assumption, on the posterior being exact, and on the costs being correct.