The earlier note on sample variance ended with a warning. When is known, the statistic has a standard normal distribution, so a probability bound for can be turned into an interval for . When is unknown, it is tempting to put in its place and keep using normal quantiles. That step is not justified. is itself random, it changes from sample to sample, and it may be correlated with . Before we can divide by and still claim an exact probability, we need the joint distribution of and .
This note supplies that missing theorem for normal samples. It then derives exact confidence intervals for and for , and it is explicit about what those intervals guarantee and what they do not.
The model
Throughout, the following assumptions are in force:
- are independent;
- each has the same distribution ;
- and ;
- and are fixed but unknown numbers.
We write
Normality is an assumption about the mechanism that produced the data. Every exactness claim below is conditional on it.
Orthogonal coordinates
The proof rests on rotating the coordinate system so that the sample mean becomes one coordinate and the spread around the mean is carried by the others.
Construction. We need an orthogonal matrix whose first row is . One explicit choice is the Helmert matrix. For , row has
- the entry in positions ,
- the entry in position ,
- zeros elsewhere.
We check that the rows are orthonormal.
- Row 1 has squared length .
- Row has squared length .
- The entries of row sum to , so row is orthogonal to row 1.
- Take rows and with . Row is nonzero only in positions . Each of those positions is among , where row has the constant entry . So the inner product is that constant times the sum of the entries of row , which is .
Hence .
Theorem 7.2.1
(Orthogonal coordinates of a normal sample)
Let and , where is orthogonal with first row . Then are independent random variables. Moreover,
Proof
(Rotation invariance of the standard normal density)
Because , each is well defined. The are independent , so has joint density
Let be any Borel set. Since , the change of variables gives
From we get . Orthogonal maps preserve length, so . Therefore has density
A joint density that factors into standard normal densities means exactly that the are independent .
The first coordinate is , and .
For the second identity, expand around the mean. Since , we have . Lengths are preserved, so . Subtracting from both sides gives
Finally, , so the right side equals .
Proof X-ray
Why the mean and the spread separate
The only probabilistic input is that the joint density of depends on through alone. Independence and normality of the observations are both used there, and nowhere else. The rest is geometry: the first row of isolates the mean in one coordinate, and Pythagoras puts the residual sum of squares in the remaining .
Independence of and then needs no calculation, because they are functions of disjoint sets of independent coordinates. Both hypotheses are needed. The counterexamples below show that normal marginals with complete dependence give , and that for independent Bernoulli observations exactly when , so the mean and the spread are dependent.
The conclusion is stronger than it first looks. depends only on , and depends only on . Independence of the coordinates therefore makes and independent. This is a special property of the normal model, not a general fact about samples.
Chi-square and Student distributions
Definition 7.2.2
(Chi-square and Student distributions)
For an integer , the chi-square distribution is the distribution of , where are independent .
If and are independent, the Student distribution is the distribution of .
The ratio is well defined with probability one, because .
Quantile notation. All quantiles in this note are lower-tail quantiles:
These definitions are unambiguous for three reasons.
-
Unique quantiles. Both distributions have continuous, strictly increasing distribution functions on their supports, so each quantile exists and is unique.
-
No atoms. Points carry zero probability. For example, using independence,
So and can be interchanged inside probabilities.
-
Positive chi-square quantiles. , so .
Symmetry of . The pair has the same joint law as , so is symmetric about . It follows that .
The two pivots
Theorem 7.2.3
(Pivots for a normal sample)
Under the model, with , define
Then , , and and are independent. Consequently
and with probability one.
Proof
(Reading the pivots off the coordinates)
By the orthogonal-coordinates theorem, and . This is a sum of squares of independent standard normals, so by definition. is a function of alone and is a function of alone, so they are independent.
Now divide. The factor cancels:
By definition this has the distribution. Finally, .
Each statistic is called a pivot because two things hold at once:
- it involves the data and the parameter of interest;
- its distribution does not depend on or .
That second property is what lets a single table of quantiles serve every normal population.
Concept lensGeometric
The t statistic as an angle
For a candidate , split into its component along , of length , and the orthogonal residual, of length . If is the angle between and the line through , then . Under the model the direction of is uniform on the sphere whatever and are, which is why is a pivot. The -interval keeps exactly the whose angle is not too small.
The intervals
Fix . Then and both lie in , and , so all the quantiles below exist and are correctly ordered.
Theorem 7.2.4
(Exact two-sided intervals)
For every fixed and every fixed ,
Proof
(Inverting the pivots)
Mean interval. Write . By symmetry and continuity of ,
On the probability-one event , multiply the inequality by the positive number . Then rearrange for :
Both events have the same probability, which is .
Variance interval. Write and , so . By continuity,
Since and almost surely, the inequality can be rearranged without changing the event. Taking reciprocals of positive quantities reverses inequalities, which gives
Common mistake
(Putting the small chi-square quantile under the lower endpoint)
In the variance interval, the larger quantile goes in the denominator of the lower endpoint. This is forced by taking reciprocals; it is not a convention you may choose. Swapping the quantiles produces a “lower” endpoint above the “upper” one, as in the exercise on supplied quantiles below.
Intervals for . The map is strictly increasing on . Taking square roots of both endpoints therefore gives an interval for with exactly the same coverage.
What exact coverage means
In the exact-interval theorem the endpoints are random variables and the parameters are constants. The statement is about the procedure. Draw a fresh sample from the same normal population, compute the interval, and repeat. In the long run, a proportion of the computed intervals contain the fixed true value. This holds exactly, for every and every .
Once the data are observed, the interval is a pair of numbers, such as . The fixed either lies in it or does not. The frequentist coverage statement does not assign probability to that particular containment. The theorem never treats or as random. "95% confidence" names the long-run reliability of the method that produced the interval.
Three quantities that are easy to confuse
| Quantity | Formula | Meaning |
|---|---|---|
| Population SD | Spread of a single observation. Unknown, and estimated by . | |
| Standard error of | (estimated by ) | Spread of the sample mean across repeated samples. |
| Half-width | Distance from to each endpoint. |
The half-width is the estimated standard error multiplied by a Student quantile. For finite that quantile is strictly larger than the normal quantile . The extra width pays for the uncertainty in .
Worked example
Worked example 7.2.5
(A five-point normal sample)
Suppose the observations are , modeled as a normal sample, so and .
Summary statistics.
- The mean is .
- The deviations from the mean are , and their squares sum to .
- So and .
- The estimated standard error is .
Quantiles. We use the following distribution quantiles as supplied numerical constants:
- ,
- ,
- .
95% mean interval. The half-width is , so
95% variance interval. Here , so
Arithmetic check by multiplication.
- .
- .
Check of the quantiles. For both distribution functions have elementary closed forms:
- : . This gives and .
- : with . At we get , , and so .
Interval for . Taking square roots gives approximately .
Notice how asymmetric the variance interval is around . With only four degrees of freedom, small values of remain quite plausible, and small corresponds to large .
Where the theorem stops
. Then is undefined and , so there is no pivot. One observation carries no internal information about spread.
. Then every , and divide by zero, and nothing above applies. The model excludes this case deliberately.
Other data mechanisms. The theorem gives no guarantee for data that are:
- non-normal,
- dependent,
- selected after looking at them,
- censored,
- collected under an adaptive stopping rule,
- heterogeneous across observations.
Each of these can break some step of the proof. Approximate results may exist in some of these settings, but they are different theorems.
Common mistake
(Using a normal quantile with the estimated )
Nothing here justifies . For and , the distribution function above, with , gives the true coverage
That is noticeably below 95%: the normal quantile ignores the variability of .
Diagnostics cannot prove normality. A normal quantile plot or a goodness-of-fit test can reveal clear departures from normality. Failing to detect a departure, however, does not establish that the data are normal. Normality remains an assumption.
Counterexample mode
Normal marginals without independence
Let , fix , and set for every .
Each is exactly , yet with probability one. As a result:
- is undefined because its denominator is zero almost surely (while its numerator is almost surely nonzero);
- the variance "interval" is , which never contains , so its coverage is .
Normal marginal distributions are not enough. The proof needs the joint density of independent coordinates.
Independence alone is not enough either: the normal shape of each observation is used as well.
Counterexample mode
A discrete sample with $n=2$
Let be independent Bernoulli, so and . Then exactly when , which has probability .
Apply the 95% recipes with , using (derived in the exercise on below), and .
When . Here and .
- The mean interval is , which contains .
- The variance interval is approximately , which contains .
When . Both intervals collapse to single points:
- the mean interval is or , which misses ;
- the variance interval is , which misses .
So each procedure has coverage exactly , far from the nominal . The chi-square and laws were never valid for this sample.
Confidence level and width
Suppose , so is the higher confidence level. Because the distribution functions are strictly increasing:
- ;
- ;
- .
Hence, for a fixed data set with , the higher-level intervals contain the lower-level ones.
Worked example 7.2.6
(Nested 90% and 95% intervals for the five-point data)
For the five-point data the estimated standard error is . With , the 90% half-width is , so the 90% mean interval is
The 95% interval uses the larger quantile and contains it. The ratio of the widths is , the same for every data set with , because both widths are multiples of the same .
Widening raises the long-run coverage of the procedure. It does not tell us whether either realized interval contains .
Confidence, prediction, and joint coverage
Prediction. A confidence interval targets the fixed number . A prediction interval targets a future observation , drawn independently of the sample.
-
Normal numerator. .
-
Independence from . is independent of , and is independent of . So the numerator is independent of .
-
Pivot. Dividing by the appropriate multiple of gives
-
Interval. The 95% prediction interval is .
Worked example 7.2.7
(Prediction interval for the five-point data)
Here , so and . The half-width is , so the 95% prediction interval is
That is far wider than the confidence interval : the ratio of half-widths is .
Under repeated iid sampling from the same normal population, the confidence-interval width tends to zero in probability as grows, while the prediction-interval width tends in probability to : a new observation retains its own spread.
Joint coverage. Let be the event that the mean interval covers , and the event that the variance interval covers . Each has probability , but the pair does not cover jointly with probability .
-
Joint coverage is below 0.95. The event has positive probability: can be extreme while . So .
-
Bonferroni lower bound. Inclusion–exclusion gives
So the joint coverage lies strictly between and . To guarantee joint coverage of at least , build each interval at level .
Exercises
Exercise 7.2.8
(Let with , where the form a normal sample.)
Core
- Show that the 95% mean interval computed from the -data is the image under of the -interval.
- Show that the variance interval scales by .
- Convert the worked example from centimeters to millimeters.
Solution(5)
- 1
The model is preserved
The are independent , and , so the exact-interval theorem applies to the -data with unchanged.
- 2
Transform the statistics
We have , and . Hence and .
- 3
The mean interval
The -interval is
As ranges over , the value ranges over exactly this set. When the endpoints swap roles, which is harmless.
- 4
The variance interval
Both endpoints contain the factor , so they scale by .
- 5
Units
Take and to convert centimeters to millimeters.
- The mean interval becomes approximately mm.
- The variance interval becomes approximately .
Variances carry squared units. That is why the variance interval scales by while the mean interval scales by .
Exercise 7.2.9
(Take , so .)
Synthesis
- Express and both intervals in terms of .
- Show that , where .
- Show that .
- Compute the 95% mean interval for the data .
Solution(4)
- 1
The statistics
With , each deviation is . So
The mean interval is .
- 2
Chi-square quantiles
If with , then for
Setting this equal to gives , hence . Applying this with and , the variance interval is
- 3
The quantile
-
Write with independent .
-
The joint density of depends only on . In polar coordinates , the density becomes . This factors, so is uniform on .
-
For , the event is the event . That set of angles has total length , so
-
Setting this equal to gives .
For : .
-
- 4
The data
Here and . The interval is
It is exact, but very wide: one degree of freedom says little about .
Exercise 7.2.10
(A normal sample of masses in grams has and . Use the supplied quantiles below.)
Core
- ,
- ,
- .
- Find the 95% intervals for , and .
- A classmate reports for . What went wrong?
Solution(4)
- 1
Mean
The estimated standard error is . The half-width is , so the interval is approximately g.
- 2
Variance
Here , so the interval is
Check: and .
- 3
Standard deviation
Taking square roots gives approximately g.
- 4
The classmate's error
The reported "interval" has its lower endpoint (about ) above its upper endpoint (about ). The classmate placed the small quantile under the lower endpoint. Reciprocals reverse order, so the large quantile must divide the lower endpoint.
Exercise 7.2.11
(Five subjects are measured before and after a treatment. The differences (after minus before) are . An analyst instead pools all ten raw measurements into one “sample.” Explain the error and give the correct 95% interval for the mean difference .)
Core
Solution(3)
- 1
Why pooling is unjustified
The ten raw values need not satisfy the model.
- The two measurements on the same subject are typically dependent.
- The before and after values may have different means.
Thus the stated information does not establish that the ten values are iid . Even if a pooled mean were estimable, it would not generally target the mean difference ; the exact-interval theorem does not justify pooling as an interval for .
- 2
The paired model
Assume the subjects are independent of one another and the differences are iid with . These are again assumptions. Then the exact-interval theorem applies to the five differences with .
- 3
The interval
These are the worked-example numbers, so the 95% interval for is approximately .
The pairing is not a nuisance to be averaged away. Differencing can remove a shared additive subject effect, but independence and normality of the differences still require their own assumptions.
Summary
For an independent normal sample with and , an orthogonal change of coordinates shows three facts:
- and are independent;
- ;
- .
These pivots give exact coverage for and for . Coverage is a long-run property of the procedure under fixed parameters. It depends entirely on the independence and normality assumptions that the proof used.