The one-sample interval answers a question about a single population mean. Many substantive questions are comparative instead: does method yield larger measurements than method , and by how much? This note builds the two-sample theory from the one-sample foundation. It also prepares the ground for contrasts and analysis of variance, where the same pooled variance reappears as a mean squared error.
Throughout, the target is the difference of population means
The estimator is almost always the difference of sample means. The real work lies in its standard error and in the distribution of the standardized statistic, and both depend on how the data were collected. Before any formula is applied, we must therefore fix the design, the sampling units, and the model.
Independent samples with a common variance
Model. Let be iid and be iid , with the two samples independent of each other, , and . Define
and similarly and . Both and are unbiased for the same . They carry different amounts of information: is built from independent pieces of residual variation, and from .
Why the pooled variance has this form. Under the model, and . If we add the residual sums of squares and divide by the total residual degrees of freedom, we obtain the following estimator.
Definition 7.4.1
(Pooled variance)
For two independent samples of sizes with sample variances and , the pooled variance is
This is a weighted average of the two variances with weights proportional to degrees of freedom. It is unbiased, because . It is exactly the error mean square of a one-way ANOVA with two groups. Writing , we have , as the proof shows.
Theorem 7.4.2
(Exact pooled two-sample interval)
Under the independent-samples model above, with ,
Consequently, for , the random interval
contains with probability exactly , for every value of with .
Proof
(Orthogonal Gaussian decomposition)
Step 1 (one group). Fix group and let be an orthogonal matrix whose first row is . Put . Because and is orthogonal, . Its first coordinate is . Orthogonality preserves squared length, so
The coordinates of are independent standard normals. Hence , , and the two are independent. The same holds for group .
Step 2 (combine). The two samples are independent, so the four quantities , , , are mutually independent. Therefore
because a sum of independent chi-squares has a chi-square law with the degrees of freedom added. depends only on the means and only on the variances, so they are independent.
Step 3 (ratio). The unknown cancels: . A standard normal divided by the square root of an independent has, by definition, the law.
Step 4 (denominator). has a continuous distribution, so . Thus almost surely, and is well defined with probability one. This step uses and .
Step 5 (inversion). The law is continuous and symmetric. Hence , and rearranging as an inequality in gives the interval.
Proof X-ray
Where each hypothesis enters the pooled t proof
Each hypothesis is used at one identifiable step. Normality is used only in Step 1: an orthogonal rotation of an vector is again , and this single fact delivers the law of the mean, the law of the residual sum of squares, and their independence. Independence between the groups is used in Step 2 to make the four statistics mutually independent.
The common variance is used in Step 2 as well, and it is the hypothesis that makes the argument close. It puts both residual sums of squares on the same scale , so their sum is times one variable and a single cancels in Step 3. With the combined residual sum of squares is , which is not a multiple of one chi-square; that is exactly the difficulty met in the Welch section.
The key move is the cancellation of . It makes a pivot, a statistic whose law under the true is free of all three parameters, and this is why the coverage is exactly for every .
What the hypotheses do and do not mean. Normality, a common variance, independence within and between fixed groups, and a fixed sample size are sufficient for exactness. They are not uniquely necessary. For example, under the true is unchanged if every error is multiplied by the same positive constant. So if the stacked error vector has any spherically symmetric distribution with finite first moments and no mass at zero, still has exactly the law, although the errors can be dependent; if their second moments exist, distinct coordinates are uncorrelated. Conversely, for markedly skewed or heavy-tailed independent errors with small samples, the coverage can differ from , and such data need separate analysis.
The theorem is a statement about sampling variability around population means. It says nothing about why the means differ. Reading as the causal effect of versus requires a design that makes the groups comparable, typically randomized assignment, or explicitly defended assumptions. Otherwise is a population association.
A worked calculation
Worked example 7.4.3
(Pooled interval for and )
Here , and , so .
Residuals in are , so . Residuals in are , so .
Pooled variance: , with .
Standard error: .
Statistic for : .
With , the margin is . The 95% interval is therefore approximately
It contains , and correspondingly .
Interpretation. The "95%" describes the procedure. Over repeated samples from the model, intervals constructed this way contain the fixed in 95% of samples. The realized interval either contains or does not; there is no 95% probability attached to it once computed.
Test–interval duality. Under the model, consider the two-sided level- test of that rejects when . This test rejects exactly when lies outside the interval, because both statements are the same inequality. The interval is the set of null values the test does not reject.
Concept lensStructural
The pooled interval as a regression coefficient
Stack all observations and fit by least squares, where for group and for group . The fitted values are the two group means, so ; the residual sum of squares divided by is ; and the usual standard error of is . The pooled interval is the regression interval for under homoscedastic normal errors, which is why its common-variance hypothesis is the regression one.
Unequal variances: the Welch interval
Now allow and with possibly . Then . The pooled formula need not estimate this variance correctly when the population variances and sample sizes differ; we use
The numerator is still normal and still independent of . The difficulty is the denominator:
This is a weighted sum of independent chi-squares with unequal weights. In general it is not a scaled chi-square, so the studentized ratio is not exactly with any degrees of freedom.
Satterthwaite's moment matching. Write the denominator as with and . Then and . We approximate by with the same two moments: set and . Solving gives . Since , substituting estimates gives the following definition.
Definition 7.4.4
(Welch interval with estimated degrees of freedom)
The Welch–Satterthwaite degrees of freedom are
defined whenever the denominator is positive, and the Welch interval is
It is approximate. Two approximations enter: the law of is replaced by a scaled chi-square, and itself is estimated from the data. There is no generic exact finite-sample coverage guarantee. Its coverage is often close to nominal for normal data, but closeness is an empirical and asymptotic property, not a theorem of exactness.
Three cautions apply.
- A preliminary test of is not a prerequisite for using Welch. Making the choice of interval depend on such a test changes the procedure's actual coverage.
- Welch is not always shorter, nor always longer, than the pooled interval. The examples below go both ways.
- Configurations such as , which have probability zero under the continuous model but occur with heavily rounded data, make the denominator vanish. Such cases and heavy-tailed nonnormal data require separate analysis.
Common mistake
(Counting pseudoreplicates as independent units)
In every case, and must count genuinely independent units. Repeated images of one specimen, or many readings from one subject, are pseudoreplicates and do not count as separate units. Entering them as separate observations inflates the sample sizes in and the degrees of freedom, and the stated coverage no longer applies; the clustered exercise at the end quantifies the effect.
Worked example 7.4.5
(Welch degrees of freedom for and )
With , Welch's , the same as the pooled value. Whenever the two standard errors coincide, because then . However,
Because quantiles decrease in the degrees of freedom, lies strictly between and . Multiplying by , the margin lies between about and , compared with the pooled margin . The approximate Welch interval is therefore wider here: with equal sample sizes it has the same standard error but fewer degrees of freedom.
Paired samples
Model. Suppose instead that the design produces pairs , iid across as bivariate random vectors, and that the differences are iid with . Independence is required across pairs. Within a pair, and may be correlated, and usually are.
Theorem 7.4.6
(Exact paired interval)
With and ,
and is an exact confidence interval for .
Proof
(Reduction to one sample)
The form one iid normal sample, so Step 1 of the earlier proof applies with replaced by . That step gives , independent of , and the ratio is . Positivity of holds almost surely because .
The point estimate is unchanged, since . What changes is the variance:
Compare this with an otherwise comparable independent design using units per arm, where . Positive within-pair covariance makes pairing more precise. Negative covariance makes it less precise.
Common mistake
(Analyzing paired measurements as two independent samples)
The independent-samples standard error assumes . In a paired design the true variance of is , and the two samples are not independent, so the pooled law does not hold. Ignoring the pairing overstates the variance when the covariance is positive and understates it when the covariance is negative. With and , the independent formula uses while the true variance is .
Worked example 7.4.7
(The same numbers, analyzed as pairs)
Suppose the design genuinely linked positions: unit produced both and , and units were sampled independently. Then . This gives , , standard error , and .
With , the margin is . The exact model-based 95% interval is approximately
This is narrower than the pooled interval despite having fewer degrees of freedom, because the within-pair differences barely vary.
Counterexample mode
Pairing is information from the design, not from the numbers
Keep and list as instead. The multiset of values is unchanged, but the pairing is different. Now . The mean is still , but , so the standard error is .
The data values alone cannot tell us which pairing, if any, is correct. Pairing is legitimate only when the sampling design links the two measurements, as with the same unit or a matched set formed before measurement. The analyst may not choose between "paired" and "independent" afterwards according to which gives a smaller -value. Doing so invalidates the stated error rates. Agreement of the numbers also does not show that the pairs are independent of one another, or that is causal.
Unequal numbers of observations per unit, missing partners, or dependence between pairs (for example, pairs measured on consecutive days that share drift) fall outside this model and need an appropriate model. A common case is comparing two fixed, already trained predictors on the same test cases. The per-case loss differences are natural pairs. But if the cases are time-ordered or clustered, for instance several records per user, the differences need not be independent across cases. In that situation the paired law is not justified merely because the same cases were used.
Exercises
Exercise 7.4.8
(Paired differences from a valid paired design are . Under the paired normal model, compute the 95% interval for .)
Foundation
Solution(2)
- 1
Summary statistics
Here and . The residuals are , so and . The standard error is with .
- 2
The interval
With ,
that is, approximately . It is exact under the stated model.
Exercise 7.4.9
(Independent normal groups have , , , and . Compute Welch’s and , describe the resulting interval, and compare it with the pooled interval.)
Core
Solution(3)
- 1
Welch standard error and degrees of freedom
, and
- 2
The Welch interval
The interval is . Its quantile lies between and and is obtained from software. The coverage is approximately, not exactly, 95%.
- 3
Compare with pooling
The pooled variance is , with , which exceeds . Here Welch’s standard error is the smaller one, because pooling gives the larger group’s larger variance extra weight. The Welch and pooled 95% margins are approximately and , so Welch’s interval is also shorter in this example, whereas in the worked example with equal sample sizes it was wider.
Exercise 7.4.10
(Valid iid pairs have and . Compare with the variance of a difference of two independent single observations.)
Foundation
Solution(2)
- 1
Within a pair
.
- 2
Independent observations
For independent observations the covariance is zero, so the variance is . With negative covariance, pairing inflates the variance by a factor of , so pairing need not improve precision.
Exercise 7.4.11
(Give a data-collection plan that is genuinely paired. Then give a clustered counterexample in which a naive individual-level interval does not have its assumed law.)
Synthesis
Solution(4)
- 1
A genuinely paired plan
Draw specimens independently from a defined population. Split each specimen into two halves, and randomize which half is measured by method . Within a specimen the two measurements are correlated through shared composition. Across specimens the pairs are independent by the sampling. Randomizing the halves supports interpreting as the effect of method on measurement for that population. If people had instead chosen a method themselves, would describe only an association.
- 2
A clustered model
Suppose five classrooms per arm, thirty students each. Let independent zero-mean classroom effects have variance , and independent zero-mean individual errors have variance . The within-class intraclass correlation is . An arm mean has variance
Because: Classroom effects and individual errors are mutually independent across classrooms and arms.
- 3
Compare with the naive model
Key step
The naive independent-student model with treats this variance as . For it ignores the factor : the true arm-mean standard deviation is about times that model-based value. This compares population standard deviations, not a realized estimated standard error. Dependence also changes the denominator distribution, so a reference and its nominal 95% coverage are not justified.
- 4
The appropriate analysis
The sampling unit is the classroom, and it must be fixed before any evaluation. If the classroom means are iid normal with a common variance (equal class sizes help), the pooled interval on the ten classroom means, with , is the appropriate analysis. Whether its is causal still depends on whether classrooms were randomized to arms.