Exact Inference in Fixed-Design Gaussian Linear Regression
From least-squares geometry to exact t intervals for linear contrasts, mean responses, and new observations, with full proofs, a computed three-point example, counterexamples, and solved exercises.
§1.7Estimated reading time: 25 min
Introduction
Least squares can be read purely as geometry: we project a data vector onto the column space of a design matrix. Confidence intervals for a normal sample can be read purely as probability: a standardized mean divided by an independent sample standard deviation has a t distribution. This note joins the two. Under a fixed design with independent Gaussian errors, the projection geometry gives three facts:
the estimator of the coefficients is exactly normal;
the residual sum of squares is an exact scaled chi-square variable;
the two are independent.
Exact t intervals then follow for nonzero linear contrasts and for a future observation.
Every observed-sample random quantity below is a function of the error vector; the later future observation also uses a separate independent error. The design is a fixed table of numbers, and all probability statements describe repetition of the errors with that table held constant.
The model and its standing assumptions
Fix positive integers n>p≥1 and a deterministic matrix A∈Rn×p with rankA=p. The parameters β∈Rp and σ2>0 are fixed but unknown. We observe
Y=Aβ+ε,ε∼N(0,σ2In).
The covariance σ2In is diagonal and ε is jointly Gaussian, so the coordinates ε1,…,εn are independent, each N(0,σ2).
Full column rank makes the p×p matrix ATA invertible. If ATAx=0 then ∥Ax∥2=xTATAx=0, so Ax=0, and therefore x=0. The same calculation shows that ATA is positive definite, and hence so is its inverse.
Definition 1.7.1
(Least-squares estimator, hat matrix and residual variance)
where we used that (ATA)−1 is symmetric. Only the mean and covariance of ε were used, not normality.
Lemma 1.7.3
(Properties of the hat matrix)
P is symmetric and idempotent with rankP=p. It satisfies PA=A, and the residual vector satisfies e=(I−P)ε.
Proof
(Five matrix identities)
Symmetry.PT=A((ATA)−1)TAT=P.
Idempotence.P2=A(ATA)−1(ATA)(ATA)−1AT=P.
Fixing the columns.PA=A(ATA)−1ATA=A.
Rank.PA=A gives col(A)⊆col(P), and P=A(⋯) gives the reverse inclusion. So col(P)=col(A), which has dimension p.
Residuals.e=(I−P)(Aβ+ε)=(A−PA)β+(I−P)ε=(I−P)ε.
The identity e=(I−P)ε is the key observation. The residual does not depend on the unknown β at all. It is the part of the noise that the column space cannot absorb.
Rotating the noise
Choose an orthonormal basis u1,…,up of col(A) and an orthonormal basis w1,…,wν of its orthogonal complement. Collect them as the columns of U∈Rn×p and W∈Rn×ν. Then Q=[UW] is an n×n orthogonal matrix, so
UTU=Ip,WTW=Iν,UUT+WWT=QQT=In.
The columns of A and of U are both bases of col(A), so A=UR for some invertible R∈Rp×p. Substituting this into P gives
P=UR(RTUTUR)−1RTUT=URR−1R−TRTUT=UUT,I−P=WWT.
Lemma 1.7.4
(Rotational invariance of the standard Gaussian)
If ε∼N(0,σ2In) and Q is orthogonal, then Z=QTε∼N(0,σ2In). In particular, Z1=UTε∼N(0,σ2Ip) and Z2=WTε∼N(0,σ2Iν) are independent.
Proof
(Through the density)
The density of ε is (2πσ2)−n/2exp(−∥x∥2/(2σ2)), which depends on x only through ∥x∥. The map x↦QTx preserves length and has Jacobian of absolute value ∣detQ∣=1. Hence Z has exactly the same density. That density factors into a product of N(0,σ2) densities, one for each coordinate, so the coordinates of Z are independent. The blocks Z1 and Z2 use disjoint sets of coordinates and are therefore independent.
The three distributional facts
Theorem 1.7.5
(Joint law of the estimator and the residual sum of squares)
Under the Gaussian model:
β^∼N(β,σ2(ATA)−1).
RSS/σ2∼χν2, and therefore E[s2]=σ2.
β^ is independent of RSS.
s>0 with probability one.
Proof
(Split the rotated noise into two blocks)
The estimator depends only on Z1. Since AT=RTUT and ATA=RTR,
β^−β=(RTR)−1RTUTε=R−1Z1.
This is a linear image of a Gaussian vector, so it is Gaussian. Its mean and covariance come from the earlier theorem.
The residual sum of squares depends only on Z2. We have e=WWTε=WZ2, so
RSS=Z2TWTWZ2=∥Z2∥2=j=1∑νZ2j2.
The terms Z2j/σ are ν independent standard normals, so RSS/σ2∼χν2. A χν2 variable has mean ν, so E[s2]=σ2ν/ν=σ2.
Independence.β^ is a function of Z1 and RSS is a function of Z2. Functions of independent vectors are independent.
Positivity.ν≥1 and Z2 has a continuous distribution on Rν, so P(Z2=0)=0.
Proof X-ray
Why one rotation gives all three facts
The single move that does the work is writing A=UR. It sends the estimator to R−1Z1 and the residual to WZ2, so each object reads only one block of the rotated noise. Full column rank is used here, to make R invertible.
Gaussianity enters only through the rotational-invariance lemma: it makes Z1, hence β^, normal, turns uncorrelated blocks into independent ones, and gives ∥Z2∥2/σ2 its chi-square law. With merely uncorrelated errors of common variance, Z1 and Z2 are still uncorrelated and E[RSS]=σ2ν survives, as shown next, but neither the independence nor the χν2 law does. The condition n>p makes the block Z2 nonempty; the chi-square law with ν≥1 and the positivity of s both rest on it.
The mean of RSS does not need normality. If only E[ε]=0 and Cov(ε)=σ2In are assumed, then
E[RSS]=E[εT(I−P)ε]=tr((I−P)σ2In)=σ2(n−p),
using (I−P)T(I−P)=I−P and trP=tr(UTU)=p. Without normality, however, the chi-square law, the independence, and the exact pivots below are not guaranteed.
Exact intervals for linear contrasts
Fix a nonzero c∈Rp and set vc=cT(ATA)−1c. Because (ATA)−1 is positive definite, vc>0 holds automatically. The target cTβ is a fixed scalar.
Write tq,ν for the lower-tail quantile: if T∼tν, then P(T≤tq,ν)=q. Write zq for the analogous quantile of N(0,1).
Theorem 1.7.6
(Contrast pivot and interval)
Tc=svccTβ^−cTβ∼tν.
For 0<α<1, the random interval
cTβ^±t1−α/2,νsvc
contains cTβ with probability exactly 1−α. If σ2 is known, the interval cTβ^±z1−α/2σvc has exact coverage 1−α.
Proof
(Standardize, then studentize)
The scalar cTβ^ is a linear image of β^, so
Z=σvccTβ^−cTβ∼N(0,1).
Z depends only on Z1, while V=RSS/σ2∼χν2 depends only on Z2, so Z and V are independent. By definition of the t distribution, Tc=Z/V/ν∼tν; the unknown σ cancels. Since s>0 almost surely, Tc is well defined.
The tν law is continuous and symmetric, so
P(∣Tc∣≤t1−α/2,ν)=1−α.
Rearranging the event ∣Tc∣≤t1−α/2,ν gives exactly the event that the interval contains cTβ. The known-variance case is the same argument with Z in place of Tc.
Common mistake
(Reading coverage as a probability about one computed interval)
The interval is random and the target is fixed. "Coverage 1−α" means the following: if independent fresh errors were drawn repeatedly with the same A, β, and σ2, the proportion of computed intervals containing cTβ would approach 1−α in the long run. A single computed interval either contains the target or does not. Coverage is also a statement made conditional on the model being true. It does not certify that a straight line, or Gaussian noise, is a correct description of the data.
Mean responses and new observations
Take a fixed row a∗∈Rp and set h∗=a∗T(ATA)−1a∗.
For a∗=0, the mean responsea∗Tβ is a nonzero contrast with c=a∗. Its interval is
a∗Tβ^±t1−α/2,νsh∗.
When a∗=0, the mean response and the displayed interval are both 0, so coverage is 1 rather than 1−α. Prediction below remains valid for this zero row. Now let Y∗=a∗Tβ+ε∗, where ε∗∼N(0,σ2) is independent of ε.
Theorem 1.7.7
(Prediction pivot)
s1+h∗Y∗−a∗Tβ^∼tν.
Consequently P(Y∗∈a∗Tβ^±t1−α/2,νs1+h∗)=1−α.
Proof
(Three independent blocks)
The numerator is normal. It decomposes as
Y∗−a∗Tβ^=ε∗−a∗TR−1Z1.
The three blocks ε∗, Z1, Z2 are mutually independent: ε∗ is independent of ε, and (Z1,Z2) is an invertible function of ε with independent blocks. Their joint density is therefore a product. The numerator is a linear combination of independent Gaussians, hence normal, with mean 0 and variance σ2+σ2h∗.
The numerator is independent of RSS. The numerator depends only on (ε∗,Z1), which is independent of Z2.
Conclusion. Divide the numerator by σ1+h∗ to obtain a standard normal, independent of RSS/σ2∼χν2. The ratio is tν exactly as in the contrast theorem.
Compare the two intervals at the same quantile. Since 1−α/2>1/2, we have t1−α/2,ν>0, and s>0 almost surely. Therefore the prediction half-width ts1+h∗ strictly exceeds the mean-response half-width tsh∗.
Common mistake
(Using the mean-response interval to predict a new observation)
The mean-response interval targets the fixed number a∗Tβ. Its probability of containing Y∗ is strictly less than 1−α, because the numerator Y∗−a∗Tβ^ has variance σ2(1+h∗), not σ2h∗. The extra 1 under the square root represents the new observation's own noise. Collecting more data cannot remove it. In contrast, h∗ typically shrinks as the design grows.
Neither interval makes extrapolation safe. The algebra is valid for any a∗, and h∗ is a positive-definite quadratic form that grows along any unbounded ray in feature space. In the intercept-and-slope example below it grows with ∣x∣, but it need not grow monotonically with Euclidean distance from the observed rows. The derivation assumes that the linear model holds at a∗; the observed design alone does not establish that assumption at a distant point.
A complete three-point example
Worked example 1.7.8
(Straight-line fit through three points)
Take n=3, p=2, rows (1,−1), (1,0), (1,1), and observed y=(0,2,2). The first column is an intercept and the second is a predictor x=−1,0,1. All responses are in the same units as y.
Normal equations.
ATA=(3002),ATy=(0+2+20+0+2)=(42),β^=(4/31).
Fit and residuals. The fitted values 34+xi are (1/3,4/3,7/3). The residuals are (0−31,2−34,2−37)=(−1/3,2/3,−1/3). They sum to zero and are orthogonal to x: 31+0−31=0.
Slope. With c=(0,1), vc=1/2 and the standard error is (2/3)(1/2)=1/3. With t0.975,1≈12.706205,
312.706205≈12.706205×0.5773503≈7.335931,
so the 95% interval is 1±7.335931=(−6.335931,8.335931). It is very wide because one degree of freedom yields a heavy-tailed t1 quantile.
Mean response at a∗=(1,2). The point estimate is 34+2=310. The quadratic form is h∗=31+24=37, and the standard error is (2/3)(7/3)=14/3≈1.247219. The half-width is 12.706205×1.247219≈15.847422, giving the interval (−12.514089,19.180755).
New observation at a∗=(1,2). The standard error is (2/3)(10/3)=20/3≈1.490712. The half-width is ≈18.941292, giving the interval (−15.607959,22.274625).
Note that h∗=7/3 exceeds 1. By contrast, the in-sample leverages Pii=31+xi2/2 are 5/6,1/3,5/6; each is at most 1, and they sum to p=2. The point x=2 lies outside the observed range [−1,1], which is exactly the extrapolation situation warned about above.
Worked example 1.7.9
(Leverage and interval widths along the fitted line)
In the three-point example, take a∗=(1,x). Then h∗=31+2x2, which grows quadratically in x and is smallest at the center x=0 of the design.
At x=0, h∗=1/3 and the point estimate is 4/3.
Mean response. The standard error is (2/3)(1/3)=2/3≈0.471405. The half-width is 12.7062052/3≈5.989762, giving (−4.656429,7.323096).
New observation. The standard error is (2/3)(4/3)=22/3≈0.942809. The half-width is ≈11.979525, giving (−10.646192,13.312858).
The ratio of prediction width to mean width is (1+h∗)/h∗. It equals 2 at x=0 but only 10/7≈1.195 at x=2. Far from the data, uncertainty about the line dominates the noise of the new observation.
So Cov(e)=6σ2wwT; for example, Cov(e1,e2)=−σ2/3=0. The orthogonal complement of col(A) is spanned by W=w/6. Hence e=6wTYw and RSS=(wTε)2/6. Here wTε/6∼N(0,σ2), so RSS/σ2 is the square of a standard normal, which is χ12: the general theorem with ν=1, seen directly.
Check with the data: wTy=0−4+2=−2, so e=−31(1,−2,1)=(−1/3,2/3,−1/3) and RSS=4/6=2/3, matching the fit above.
The residuals satisfy e2=−e1, and e1∼N(0,σ2/2) has positive variance because σ2>0. Hence P(e1>0,e2>0)=0, while P(e1>0)P(e2>0)=1/4, so the components are dependent.
In general Cov(e)=σ2(I−P) has nonzero off-diagonal entries. Independence of the errors does not pass to the residuals.
Two further boundary cases fall outside the standing assumptions altogether.
Counterexample mode
Rank deficiency and the case $n=p$
Rank deficiency. Let the columns of A be (1,1,1)T and (2,2,2)T. Then ATA=(36612) has determinant 36−36=0, so β^ is undefined. Moreover, β and β+t(2,−1) produce the same mean Aβ for every t. No data can distinguish them, and the slope c=(0,1) has no meaningful interval in this framework.
The case n=p. With full rank, col(A)=Rn, so P=I and e=0 for every sample. Then RSS=0, ν=0, and s2=0/0 is undefined. The fit is exact, and the residuals supply no degrees of freedom for this variance estimator.
Both cases lie outside the standing assumptions and must be treated separately.
The fixed-design assumption also matters. Every probability above is computed with A held constant. Statements about designs drawn at random, about errors with unequal variances, about intervals reported after choosing variables or checking diagnostics, about distribution-free robustness, and about several intervals holding simultaneously would each need separate arguments. Nothing here addresses them. A coefficient interval also measures association within the model; it does not by itself establish a causal effect.
Exercises
Exercise 1.7.11
(Let A=1n with n≥2. Derive β^, P, RSS, the interval for β, and the prediction interval.)
Foundation
Solution(4)
1
Compute the estimator and the projection
ATA=n and ATY=i∑Yi, so β^=Yˉ. Also P=n111T, so ei=Yi−Yˉ and RSS=i∑(Yi−Yˉ)2.
2
Identify the variance estimate
ν=n−1, so s2 is the usual sample variance.
3
Apply the contrast theorem with c=1
vc=1/n, and the contrast theorem gives (Yˉ−β)/(s/n)∼tn−1. The interval is Yˉ±t1−α/2,n−1s/n, which is the one-sample t interval.
4
Apply the prediction pivot with a∗=1
h∗=1/n, so the prediction interval is Yˉ±t1−α/2,n−1s1+1/n.
Exercise 1.7.12
(For λ=0, compare the intervals for cTβ and (λc)Tβ. Then evaluate the case λ=−3 with the slope in the three-point example.)
Core
Solution(3)
1
Rescale the standard error
Replacing c by λc multiplies vc by λ2, so the standard error becomes ∣λ∣svc.
2
Rescale the pivot
Key step
The pivot becomes sign(λ)Tc, which has the same tν law and the same absolute value. So the new interval is λcTβ^±t∣λ∣svc. This is the old interval multiplied by λ, with endpoints swapped when λ<0.
3
Evaluate c=(0,−3)
The estimate is −3 and the standard error is 3/3=3. The half-width is 12.7062053≈22.007792, giving the interval (−25.007792,19.007792). Check: multiplying (−6.335931,8.335931) by −3 gives approximately (19.007792,−25.007792), the same interval with its endpoints reordered.
Exercise 1.7.13
(Fit A with rows (1,0) and (1,1) to y=(1,3), and explain what fails.)
Foundation
Solution(2)
1
Solve the square system
A is invertible, so β^=A−1y=(1,2), and the fitted values (1,3) reproduce the data exactly. Hence e=0, RSS=0, and ν=0.
2
Locate the failure
s2 is undefined and there is no tν law, so no interval of this form exists. The data determine the fitted line, but the zero residual degrees of freedom do not provide the variance estimate needed for this t procedure.
Related notes
Prerequisites are orthogonal projection and least squares, multivariate normal vectors, and the chi-square and t distributions for normal samples.