Ordinary least squares fits a model by orthogonal projection. A scalar contrast test asks whether one linear combination of coefficients is zero. Many questions, however, involve several restrictions at once. Are three group means equal? Do two extra predictors add anything beyond a baseline model? The nested regression test answers questions of this form. One-way analysis of variance (ANOVA) is its most familiar special case.
This note derives the exact null distribution of the statistic from projection geometry. It then specializes to one-way ANOVA and states which assumptions the conclusion depends on.
Setting and assumptions
Fix two design matrices and with
Both matrices have full column rank. When we take . We observe
Here is an unknown fixed vector.
Several words in this setting carry real weight.
- Fixed. The matrices , and therefore the two models, are chosen before the responses are seen. They do not depend on .
- Nested. The reduced model space lies inside the full model space .
- Full column rank. This makes invertible. If a design you want to use is rank deficient, either delete redundant columns, or replace throughout by . The theory below then holds with , provided and . Counting columns instead of rank can make a degree of freedom vanish, and that leads to division by zero.
Definition 1.8.1
(Projections and residual sums of squares)
Let be the orthogonal projection onto , with when . The residual sums of squares are
The second equality uses .
Projection geometry
Lemma 1.8.2
(Nested projections)
Let . Then:
- and are symmetric idempotent matrices, so each is an orthogonal projection.
- , so the two projections project onto mutually orthogonal subspaces.
- and .
Proof
(Products, traces and the drop in RSS)
Every column of lies in , and fixes every vector of . Hence . Transposing and using symmetry gives .
Symmetry of and is immediate. For idempotence,
The same kind of calculation gives . For orthogonality,
For an orthogonal projection, rank equals trace. By cyclicity of the trace,
Therefore and . Finally,
Since is a symmetric idempotent matrix, this equals .
Part 4 has a practical consequence. Enlarging a model can never increase the training RSS. A drop in RSS is therefore guaranteed, and only its size, measured against noise, carries evidence.
The exact null distribution
Theorem 1.8.3
(Nested test)
Suppose (the null hypothesis ). Then:
- ,
- ,
- these two quantities are independent, and almost surely.
Consequently
Proof
(An orthonormal basis adapted to the nested models)
Step 1: an adapted orthonormal basis. Choose an orthonormal basis of . Extend it by to an orthonormal basis of . Then extend by to an orthonormal basis of . With this basis,
Let . This matrix is orthogonal.
Step 2: remove the mean. Under we have . Hence and . So
Step 3: rotational invariance. Since , the vector is Gaussian with covariance . Thus are independent variables. Because ,
Step 4: read off the laws. The first sum has independent squared standard normals, so it is . The second has , so it is . The two sums use disjoint sets of independent coordinates, so they are independent.
Step 5: the denominator is positive. Since , . The event has probability zero. Hence almost surely and is well defined.
Step 6: form the ratio. The ratio of two independent chi-square variables, each divided by its degrees of freedom, has the distribution by definition. The unknown cancels from the ratio.
Proof X-ray
Where each hypothesis enters
The decisive move is Step 1. Once the basis is adapted to the nested pair of subspaces, both sums of squares become sums of squared coordinates of a single rotated noise vector, over disjoint index sets. Independence and both chi-square laws then follow at once from rotational invariance; no separate independence argument is needed.
- Step 2 needs . The full model must contain the true mean, and under the null so must the reduced model.
- Step 3 needs independent, homoscedastic Gaussian errors. That is exactly what makes rotationally invariant.
- Step 5 needs and .
Behavior under alternatives. Suppose instead that but . The denominator is still . The numerator, however, becomes times a noncentral variable with noncentrality . Then follows a noncentral distribution, which tends to produce larger values. That is why the test rejects in the upper tail.
If , even is noncentral. In either case it would be wrong to claim the central law for arbitrary . The central law is a statement about the null only.
Scale invariance. Replace by with . Both residual sums of squares are multiplied by , so does not change. In particular, the test does not depend on the units of measurement.
Concept lensGeometric
The F statistic as an angle
The reduced-model residual splits orthogonally into and . If is the angle between and the added subspace , then and , so . The test rejects when points nearly into the added directions. In one-way ANOVA, , the familiar ; in the three-group example below, and .
The case . Suppose the reduced model imposes a single fixed linear restriction on the full model . Write and .
Set . This vector lies in . It is also orthogonal to every with , because . Hence is the projection onto , and
Dividing by shows that , where
The upper-tail test is therefore the two-sided test for the same fixed contrast. It is not a one-sided test, because squaring discards the sign of .
One-way ANOVA as a nested comparison
Suppose groups are nonempty, with sizes and total . The observations are
All groups share the same , but the means may differ.
The full model uses the indicator matrix . Its columns have disjoint, nonempty supports, so has full column rank and has rank . Projecting onto replaces each observation by its group mean .
The reduced model uses . This is the sum of the columns of , so the models are nested, and has rank . Projecting onto replaces every observation by the grand mean
This is a weighted average of the group means, with weights proportional to the group sizes.
Common mistake
(Using the unweighted mean of the group means)
With unequal group sizes, differs from . It is not the mean of the pooled data, and it is not the projection of onto , so it does not produce . Replacing by in adds , which is not between-group variation, and then fails. The last exercise measures the discrepancy in a small case.
The two residual sums of squares are the familiar ANOVA quantities:
Proposition 1.8.4
(ANOVA decomposition)
Define . Then . Under ,
Proof
(The cross term vanishes)
Write each deviation from the grand mean as a within-group part plus a between-group part:
Squaring and summing gives
The inner sum is zero for every , because deviations from a group's own mean sum to zero within that group. So the cross term vanishes.
For the distribution, says precisely that . The nested theorem applies with and .
Worked example 1.8.5
(Three groups of two)
Take groups , and , so that and .
Means. The group means are . The grand mean is .
Within-group sum of squares. Every observation lies one unit from its group mean, so each group contributes . Hence .
Between-group sum of squares.
Total sum of squares. The deviations from are . Their squares sum to , as the decomposition requires.
Test statistic. The degrees of freedom are and . So
p-value. When the numerator has degrees of freedom, the tail probability has a closed form. Here is a standard exponential variable . Write . Then
At this gives .
Conclusion. At level we do not reject .
Common mistake
(Reading non-rejection as equal means)
Failing to reject does not establish that the means are equal. The sample means visibly differ; the power of the test depends on the true mean differences, variance and design, not on alone. The p-value is the probability, computed under , of an value at least as large as the one observed. It is not . A probability of that kind requires a prior and a Bayesian model.
In the other direction, a significant value says that the group-specific model fits better than a single common mean by more than noise would explain. Whether the groups cause the difference depends on how units came to be in the groups. Random assignment supports a causal reading. Observational grouping allows confounders that the test cannot detect.
With two groups, ANOVA is an instance of the identity.
Worked example 1.8.6
(Two groups: the statistic is )
Take groups and , so that and .
- The group means are and , and the grand mean is .
- Each observation lies one unit from its group mean, so .
- .
Therefore
For the pooled two-sample statistic, the pooled variance is . The difference in means is , with standard error . So and .
The contrast was fixed in advance, so the upper tail of matches the two-sided test. Using the distribution function, the common p-value is .
Worked example 1.8.7
(Counting columns instead of rank)
Return to the groups , , , but write the full model with an intercept column and all three indicators. That matrix has columns but rank , because the indicators sum to the intercept. Its column space is still the space of group-mean vectors, so the orthogonal projection onto it, and are unchanged; only the formula with is unavailable, because that matrix is singular.
With ranks, and , which recovers . Counting columns gives and , hence referred to : a different statistic and a wrong reference law.
The degenerate case is sharper. For groups and , , and the intercept-plus-indicators matrix has columns but rank . Column counting gives and a division by zero. With ranks, : the group means are and , the grand mean is , , , and on degrees of freedom.
Counterexample mode
Zero within-group variation
Consider the observed data , . The group means are and , the grand mean is , and . The formula for divides by .
One might try to model this by setting . In that case is deterministic and the theorem does not apply.
- Under the null, numerator and denominator would both be zero, giving .
- For the depicted unequal-group data, the expression would be .
- In neither case is the expression -distributed.
Within a valid Gaussian model with , the event has probability zero. So such data are a signal about the model, not a test result. Typical causes are rounding, a deterministic measurement process, or a mis-specified error structure. The response is to revisit the model. Reporting an infinite is not an appropriate response.
What the reference distribution does and does not guarantee
The assumptions in the theorem are sufficient for the reference; other special settings may have the same law for different reasons.
- Models fixed in advance. If you try many candidate models and report the one with the largest , the reported statistic is a maximum over many statistics. It need not follow , and the naive p-value can be anti-conservative. The projection argument assumes and do not depend on .
- Independent, homoscedastic Gaussian errors. Under unequal variances or dependence, is no longer standard normal. The quoted reference is then not guaranteed, even approximately in small samples.
- The full model contains the truth. If it does not, the expected residual mean square is , greater than ; an individual observed residual mean square need not exceed it.
- The question being answered. compares two nested subspaces in-sample. It does not measure out-of-sample prediction accuracy, it does not establish causation, and it is not a posterior probability.
Exercises
Exercise 1.8.8
(With , , , and , compute and interpret it.)
Foundation
Solution(3)
- 1
Compute the statistic
Here and , so
- 2
Use the closed form for two numerator degrees of freedom
The same closed form as in the three-group example gives
- 3
Interpret the drop
The drop from to is not significant at conventional levels. The drop alone carries no evidence, because the lemma guarantees for any data. Evidence comes only from comparing the drop per added dimension, , with the estimated noise per residual dimension, .
Exercise 1.8.9
(An analyst has candidate predictors and compares the two procedures below. When is the reference exact?)
Core
- (a) Before collecting data, they fix a model with specific predictors and test it against an intercept-only model.
- (b) They look at the outcomes, keep the predictors most correlated with , and then run the same test.
Solution(3)
- 1
Procedure (a): fixed subspaces
In (a), and are fixed. If the intercept and four predictors have full column rank, , the errors are independent with , and lies in the intercept-only space, then exactly.
Because: The theorem requires and chosen without reference to .
- 2
Procedure (b): a data-chosen subspace
Key step
In (b), is a function of . The projection proof assumes a fixed subspace; choosing predictors using violates that premise. The usual reference distribution is not generally guaranteed, and selection can make the naive p-value anti-conservative. A valid analysis would need, for example, a held-out sample or a method that accounts for the selection step.
- 3
What even (a) does not show
Even in (a), a small p-value says only that the larger subspace explains more variation than noise would. It does not say the model predicts new data well, and without randomization it does not identify causal effects.
Exercise 1.8.10
(Prove that , and give an equivalent expansion. Then compute for group means and with and .)
Core
Solution(3)
- 1
Sum a term that does not depend on
The summand does not depend on . Summing it over the observations in group multiplies it by , which proves the first identity.
- 2
Expand the square
Because: .
- 3
Compute with the weighted grand mean
The grand mean is the weighted mean . Then
The expansion agrees: . With the unweighted average in place of , the formula would give . It exceeds by , so the decomposition no longer holds.