Which population is in the denominator?
Definition 7.1.1
(Precision, recall and false positive rate)
For a binary classifier, write for true positives, false negatives, false positives and true negatives. We use rows for the actual class and columns for the predicted class. Write for precision, for recall and for false positive rate. If the relevant denominator is positive,
Precision conditions on a positive prediction; recall conditions on an actual positive; the false positive rate conditions on an actual negative. If there are no positive predictions, precision has denominator zero and is undefined under these formulas. Likewise, recall and false positive rate require actual positives and actual negatives respectively. A reporting convention may choose a replacement, but must identify it rather than treating it as the ratio above.
This unit describes evaluation on labeled populations.
Two populations with the same conditional rates
Worked example 7.1.2
(Same recall and false positive rate, different precision)
Consider the following counts for the same fixed decision rule. The columns are predicted positive and predicted negative, respectively.
| Population | Actual class | Predicted + | Predicted − |
|---|---|---|---|
| A | Positive | 30 | 10 |
| A | Negative | 20 | 140 |
| B | Positive | 30 | 10 |
| B | Negative | 45 | 315 |
Population A has observations and prevalence ; population B has and prevalence . In both, recall is and the false positive rate is . Yet their precisions differ:
Changing the relative frequency of actual classes changes the mixture among predicted positives even when both conditional rates stay fixed. These are constructed counts, not a claim that a real classifier will preserve its rates when deployed elsewhere.
Proposition 7.1.3
(Precision from the base rate)
Let , and , with and . Then
Proof
(Partition by actual class)
Partitioning by the actual class gives
and the joint event of an actual and predicted positive has probability . The definition of conditional probability divides the second by the first, which is the stated formula. For fixed and , substitution of or yields the two precisions and above.
The formula is a conditional-probability identity, so no causal or independence assumption is involved.
Proof X-ray
What the base rate does and does not change
The proof has one move: split the predicted positives by the actual class they come from. The two classes need positive probability so that and are defined, and so that the final division is legitimate; this last condition is exactly the denominator condition in the definition of precision.
The prevalence enters only as the weight on each term of the denominator. Recall and false positive rate are conditional on the actual class, so they do not see , while precision, which conditions on the prediction, mixes the two classes by their frequencies. The formula holds for each population separately; it does not say that and will stay fixed when the population changes, which is a modeling claim the proof does not make.
Worked example 7.1.4
(Counts from prevalence and rates)
A population of has prevalence , and a rule has and . There are actual positives and actual negatives. Then , , and . Precision is . The formula agrees: and , so . Although the rule finds of positives and has only a false positive rate, fewer than a third of its positive predictions are correct, because actual negatives are so much more common.
Worked example 7.1.5
(A rule that predicts positive for everyone)
Use population B, which has actual positives and actual negatives, and predict positive for all observations. Then , , . Recall is , the false positive rate is , and precision is , which equals the prevalence. Perfect recall is therefore compatible with the lowest precision that any rule with perfect recall can have on that population. The opposite edge case, predicting negative for everyone, has , so recall is , the false positive rate is , and precision is , undefined.
Thresholds and a common trap
Suppose a fixed labeled evaluation set has a real score for each observation, and predict positive when the score is at least a threshold . Raising can only remove predicted positives. If the set contains at least one actual positive, cannot increase, so recall cannot increase. The false positive rate is also nonincreasing when actual negatives exist. Precision, however, need not move monotonically: its numerator and denominator both change.
Counterexample mode
A higher threshold need not improve precision
Take four observations with score and actual label . With the inclusive rule above, thresholds , and give precision , then , then . Raising the threshold first lowers precision and then raises it. These calculations use a fixed evaluation set; they do not establish a population-level performance guarantee.
Common mistake
(Assuming precision follows from recall and false positive rate)
Populations A and B share recall and false positive rate , yet their precisions are and . Precision cannot be read from the two conditional rates alone; the prevalence is also required.
Common mistake
(Dividing by the wrong total)
In population A, is recall, not precision. Precision divides by the predicted positives, . Always ask which row or column total sits in the denominator.
Exercises with solutions
Quick check
(In population A, what fraction of actual positives are missed?)
Use the actual-positive row.
Solution · Answer
.
Quick check
(A rule predicts negative for everyone. What is its precision under the displayed definition?)
Compute first.
Solution · Answer
Undefined, because ; a separate reporting convention must be stated.
Quick check
(If , and , what is precision?)
Use the formula from the base rate.
Solution · Answer
.
Quick check
(Why specify whether a score equal to the threshold is positive?)
Think about which confusion counts a tied observation affects.
Solution · Answer
A tied observation can change the confusion counts, so the evaluation is not determined without that rule.