Many quantities reported in data analysis are proportions. Examples include the fraction of held-out negatives that a classifier flags as positive, the fraction of positives it detects, and the fraction of transactions that fail. A single number such as does not show how well the underlying rate is determined. These notes develop three interval procedures for a binomial proportion. They explain what each one does and does not guarantee, and they show why the most familiar formula is unreliable in the cases where an interval matters most.
The model and what coverage means
Let be independent Bernoulli() random variables, with fixed and unknown. The count has the binomial distribution
At or , interpret the formula by continuity (equivalently, take ). We observe and write . Fix .
This model contains assumptions that should be checked rather than taken for granted. Independence fails, for example, when several test images come from the same patient. A common fails when the units are a fixed collection with different individual event probabilities; the count is then Poisson-binomial rather than binomial. If the units are drawn independently from a population, each is Bernoulli with the population rate even when the population is heterogeneous. Heterogeneity is therefore not automatically a problem, but the sampling design has to justify the model. In classification, might be a false positive rate: the probability that a fixed classifier flags a randomly drawn negative case.
Definition 7.3.1
(Confidence procedure, coverage and confidence level)
A confidence procedure is a pair of functions on . Its coverage at is
The procedure has confidence level if for every .
In this probability, is fixed and the endpoints are random because is random. Once is observed, the interval is a fixed set of numbers. It either contains or it does not.
Common mistake
(Reading the confidence level as a probability about the computed interval)
The statement “there is a 95% probability that lies in ” assigns a probability distribution to , which this framework does not do. The 95% describes the long-run behavior of the procedure under repeated sampling from the model. The computed interval either contains the fixed number or it does not.
From the normal-mean interval to the binomial
If are independent with known, then is exactly standard normal. Let . Then has coverage exactly .
For Bernoulli data, has mean and variance . By the central limit theorem, for and large ,
Two differences from the normal case matter. The variance depends on the unknown , and is discrete, so the normal approximation is never exact. There are two natural ways to handle the unknown variance. Replacing by in the variance gives the Wald interval. Keeping in the variance and solving for the set of plausible gives the Wilson interval.
The Wald interval and its boundary failure
Definition 7.3.2
(Wald interval)
With , the Wald interval for is
When or , the estimated standard error is and the interval collapses to the single point or . A zero estimated standard error does not show zero uncertainty. It is the plug-in formula evaluated at a boundary value of , where the formula carries no information about uncertainty. The Wald interval can also leave . With and , the half-width is , which gives a lower endpoint of about .
Common mistake
(Clipping the Wald interval to )
Intersecting the Wald interval with removes the impossible values, and it is tempting to regard this as a repair. It cannot change the coverage at all. For every and every interval , the statement holds exactly when , so the clipped procedure has the same coverage function as the original. The calibration problem is untouched.
Counterexample mode
Wald coverage can be zero
Take . If , the Wald interval is ; if , it is . For any with , neither interval contains , so . The procedure is labeled "" for every (in particular, 95% when ), yet its coverage is at every interior . Coverage equals only at and . The defect is less extreme for larger , but it persists near the boundary whenever is small.
The Wilson score interval
The score approach keeps the null-hypothesis variance. For , a value is retained when
At or the standardized statistic is undefined. With and , for instance, it has the form . To avoid this, define the retained set using the squared form, which involves no division:
For interior the two conditions are equivalent. At the squared condition reads , so is retained exactly when . This matches the limit of the interior set as .
Theorem 7.3.3
(Wilson score interval)
For every , the set is the closed interval with center and half-width
Both endpoints lie in . The lower endpoint is exactly when , and the upper endpoint is exactly when .
Proof
(Solving the quadratic and locating its roots)
Expanding and collecting terms gives , where
The discriminant is
so has two distinct real roots . Because the leading coefficient is positive, exactly on . The quadratic formula gives roots with and as stated; the factor from the square root of the discriminant cancels against the in the denominator.
To see that the roots lie in , evaluate at the endpoints:
The vertex lies strictly inside . Clearly , and is equivalent to , which always holds. Since and , the point is either outside to the left, when , or equal to , when . In both cases , with equality iff . The same argument at shows that , with equality iff .
For , the formulas reduce to , so the interval is . By the symmetry , the interval for is .
The proof shows that no statistical clipping is needed. If floating-point arithmetic returns a lower endpoint of when , replacing it with only corrects rounding error.
The Wilson interval inverts an approximately calibrated test. Its coverage function is not guaranteed to stay above for any finite . Because takes only values, the coverage jumps as crosses an endpoint. For small counts, the coverage can fall noticeably below nominal at particular values of , as the first exercise below shows.
Exact equal-tailed tail inversion
For a guaranteed level, the normal approximation can be replaced by the binomial tails themselves.
Definition 7.3.4
(Exact equal-tailed interval)
Set . For , let be the unique with . Set . For , let be the unique with . The interval is commonly called the Clopper–Pearson interval.
Existence and uniqueness of these roots are part of the next theorem.
Theorem 7.3.5
(Binomial tails are strictly monotone in )
For , the function is continuous and strictly increasing on , with and . For , the function is continuous and strictly decreasing on , with and . Consequently, and are well defined.
Proof
(A coupling argument)
Both functions are polynomials in , so they are continuous. Let be independent Uniform variables and define . Then , and pointwise whenever . Therefore
For , consider the event that and . On this event and . The event has probability , so is strictly increasing on . By continuity, it is strictly increasing on .
At the endpoints, because and almost surely, and . Since , the intermediate value theorem gives a root, strict monotonicity makes it unique, and the endpoint values exclude . The result for follows from .
The interval is also nondegenerate. For , at we have . Since is decreasing, this gives .
Concept lensAnalytic
Exact endpoints as quantiles of order statistics
The coupling gives more than monotonicity. In it, holds exactly when the th smallest of is at most , so is the distribution function of that order statistic, which has the Beta law. Hence is the quantile of Beta and, since , is the quantile of Beta. For this is , as in the zero-event example.
Theorem 7.3.6
(Guaranteed coverage)
For every , .
Proof
(Super-uniform tail probabilities and a union bound)
At , almost surely and contains . At , almost surely and contains . In both cases the coverage is .
Now fix and let , which is nonincreasing in . Regard as the upper-tail p-value for testing this .
Super-uniformity. For , the set is either empty or of the form , because is nonincreasing. Hence or . In either case, .
Lower-endpoint failure. Suppose . Then , since . By strict monotonicity in , , where denotes a fresh copy. Therefore .
The same argument applies to , which is nondecreasing in , and gives . By the union bound, the probability of non-coverage is at most .
The failure conditions are strict inequalities. If exactly, then , and the closed interval still contains . If there is a with , let be the smallest such ; then . If there is no such , the failure event is empty and its probability is . Hence for interior the coverage is in fact strictly greater than , although the excess changes irregularly with .
Proof X-ray
Two monotonicities and no approximation
The coverage proof uses monotonicity in two different variables. Monotonicity of in makes the p-value super-uniform: is the largest attainable tail probability not exceeding (or ), never more than . Monotonicity of the tail in , which is the coupling theorem, is what turns the endpoint event into the p-value event at the true . Without that link the definition of , made at the parameter value , would say nothing about probabilities at .
The union bound adds the two tails. Here it loses nothing, because makes the events and disjoint. The conservativeness comes entirely from discreteness: is usually not an attainable tail value, so each tail contributes strictly less. No normal approximation, and no condition on or , appears anywhere.
In this context, "exact" means that the procedure uses exact binomial probabilities and therefore has a guaranteed level. It does not mean that the coverage equals . Because of discreteness, the coverage typically exceeds the nominal level, so the procedure is conservative.
Computing the endpoints does not require special-function software. and are explicit polynomials and strictly monotone, so bisection on converges to the unique root. A solver stops at some tolerance , so the reported endpoint is only known to within . To keep the guarantee, round the lower endpoint down and the upper endpoint up.
Worked comparisons
Worked example 7.3.7
(Zero events in ten trials)
Take , , and , so . Suppose ten held-out negatives produced no false positives.
Exact. . solves , so . Since , we get . The interval is .
Wilson. .
Wald. .
The three procedures give genuinely different answers. Both the Wilson and exact intervals indicate that false positive rates up to roughly 28–31% are compatible with these data. The Wald interval, which suggests a rate of exactly zero, reflects a failure of the procedure and carries no information about the classifier. None of these intervals carries a probability statement about . Each is a computed set of numbers, and the guarantee, exact or approximate, belongs to the procedure that produced it. The guarantee also depends on the binomial model being correct.
By the symmetry , the case gives exact lower endpoint , which is here, and Wilson lower endpoint .
Worked example 7.3.8
(Two events in four trials)
Take and . Then
So solves in . The left side is increasing; its derivative is . At it equals about , slightly too large. One Newton step with derivative gives , and at that value the polynomial equals to six decimals.
For the upper endpoint, if then . Thus , and .
For comparison, the Wilson interval is centered at with half-width , giving . The Wald interval is . Here Wald is wider than the exact interval, so width alone does not indicate whether a procedure is calibrated.
Worked example 7.3.9
(One event in ten trials)
Take and , so , with and .
Wald. The half-width is , so the interval is , with a negative lower end.
Wilson. Here , so and
The interval is . It lies inside , as the Wilson theorem guarantees, and it is not centered at .
Exact. solves , so . Since , we get and . solves , which has no closed form; bisection gives . Check: at , .
The exact interval is the widest of the three. Its lower end is tiny because one event in ten trials is not rare even for small : at the probability of at least one event is still .
One-sided bounds and many metrics
For , a one-sided upper bound solves ; set , since that equation has no solution when . This is a different procedure from the two-sided interval. The upper endpoint of the equal-tailed interval is a upper bound. If only an upper bound on an error rate is of interest, the one-sided procedure is the appropriate one, and the choice should be made before seeing the data.
When several intervals are computed from one evaluation, each carries its own marginal guarantee. Nothing in the coverage theorem implies that they all hold at once.
Exercises
Exercise 7.3.10
(For , find the coverage function of the Wald, Wilson and exact procedures at .)
Foundation
Solution(3)
- 1
Wald
As shown in the counterexample above, for and at .
- 2
Wilson
Key step
With , the intervals are for and for . So for , on , and for . At , the coverage is , confirming that Wilson is only approximately calibrated.
Because: The intervals for and are and .
- 3
Exact
and . Hence for , on , and above that. Every value exceeds .
Exercise 7.3.11
(Derive closed forms for the exact endpoints when and . Evaluate them for , , and compare with the one-sided 95% upper bound when .)
Core
Solution(4)
- 1
Closed forms
For : , so and . For : , so and .
- 2
Evaluate at
. This gives for and for .
- 3
The one-sided bound
The one-sided bound solves , so . This is smaller than because all of is placed in one tail.
- 4
Check
Since for , we have , and the computed value satisfies this.
Exercise 7.3.12
(Show that the Wilson interval has width at most for every . Find the smallest guaranteeing width at most at , and compare with the corresponding Wald calculation.)
Core
Solution(3)
- 1
Bound the width
The width is . Substituting ,
with equality at .
Because: increases with , which is at most .
- 2
Smallest sufficient
Requiring gives , so . Check: , whereas gives about .
- 3
Compare with Wald
This bound holds for every possible , so it is a deterministic statement about width; it says nothing about coverage. The Wald worst case requires . At , the Wilson width shrinks like , much faster than the worst case.
Exercise 7.3.13
(A fixed classifier, chosen independently of its test data, is evaluated on an i.i.d. test sample of labeled cases with a predetermined subgroup indicator. The report gives exact 95% intervals for sensitivity (among positives), specificity (among negatives) and the error rate in the subgroup. The relevant population probabilities are positive. If a metric has no eligible cases in the sample, report for it. What is guaranteed?)
Synthesis
Solution(4)
- 1
A binomial model for each metric separately
For each metric, condition on which cases are eligible for that metric. By i.i.d. sampling, their outcomes are independent Bernoulli variables with that metric’s common population rate; for example, the positive cases have the same marginal sensitivity. Do not additionally condition on subgroup membership when analyzing overall sensitivity: performance may differ by subgroup.
Because: The binomial model follows from i.i.d. sampling and the fixed classifier, not from independence alone.
- 2
Unconditional coverage of each interval
Given a positive eligible count, exact tail inversion has conditional coverage at least ; with count zero, covers automatically. Averaging over that metric’s eligibility configurations gives each interval unconditional coverage at least .
- 3
Simultaneous coverage
Key step
Each interval fails with probability at most , so all three cover simultaneously with probability at least . The figure is not justified because the intervals need not be independent. For a family-wise level of , use exact intervals at level for each metric.
Because: Union bound over three failure events.
- 4
What the 95% is not
Let be the number of the three intervals that cover their targets. By linearity of expectation, , but the realized fraction is one of or . The 95% is therefore not the proportion of intervals in this report that are correct. It is also unrelated to prevalence: the class proportion is a separate parameter, and it would need its own interval.
Summary
The Wald interval substitutes into the variance. It collapses at and and can have arbitrarily poor coverage. The Wilson interval inverts the score test by solving a quadratic. Its endpoints always lie in , but its calibration is only approximate. Exact tail inversion guarantees coverage of at least at every , at the cost of conservativeness caused by discreteness. All three guarantees concern repeated sampling from a stated model, hold one interval at a time, and depend on the independence and common-rate assumptions that justify the binomial model.