Why resample at all?
Earlier notes on sampling risk treated the sample mean as a random variable. If are independent and identically distributed with mean and finite variance , then . The standard error describes how far typically falls from across hypothetical repetitions of the whole data-collection process. Confidence intervals were built from this quantity.
Two difficulties remain. First, is a property of the population and is unknown. Second, for many statistics, such as medians, trimmed means, ratios, or the accuracy of a fitted classifier, no simple variance formula is available. The nonparametric bootstrap addresses both by building an artificial sampling experiment out of the data we actually have.
The sample mean is the right place to study the method. For the mean every bootstrap quantity can be computed exactly. We can therefore see what the bootstrap estimates, how it relates to the classical standard error, and where it can fail.
Two layers of randomness
Fix the observed real numbers with . Write
The quantities and are both computed from the same sum of squares. They differ only in the divisor, so .
Definition 7.7.1
(Empirical distribution and bootstrap sample)
The empirical distribution places mass on each observation; repeated values receive accumulated mass. A bootstrap sample is obtained by drawing indices independently and uniformly from and setting . Equivalently, we draw values with replacement from . The bootstrap mean is .
The star notation , , means probability, expectation, and variance with respect to the resampling only, conditional on the data. The numbers are held fixed. The only thing that varies is which indices are drawn. This conditional world must be kept separate from the original sampling world, in which the data themselves are random.
Two different counts also appear, and they must not be confused:
- is the size of each bootstrap sample. It equals the original sample size, because we want to mimic a statistic computed from observations.
- is the number of bootstrap samples we draw on a computer. It controls Monte Carlo accuracy and has nothing to do with how much information the data contain.
Exact conditional moments
Theorem 7.7.2
(Conditional moments of the bootstrap mean)
Conditional on the observed data, for each ,
and
Proof
(Computing through the resampled index)
Since is uniform on , the draw satisfies
Working with indices rather than distinct values handles ties automatically: a value appearing twice simply contributes twice.
For the variance,
The divisor is , not . This is because is being treated as an exact population with known mean . Nothing is estimated inside the resampling world, so no degrees-of-freedom correction appears.
Linearity gives . The indices are independent, so the are conditionally independent and their variances add:
Substituting gives the final form.
The bootstrap standard error of the mean is therefore . It is a function of the data alone. No population parameter appears, and none could, because the resampling world knows only .
Concept lensCombinatorial
A bootstrap sample as a vector of counts
Only how often each index is drawn matters. Let count the draws of index ; then is multinomial with trials and equal probabilities , and . From and for one recovers . There are count vectors, so for there are three, matching the three values of in the example below.
Worked example 7.7.3
(The dataset )
Take with , . Then . The deviations are and , so the sum of squares is . This gives and .
A bootstrap sample is an ordered pair of draws, and each of the four pairs has probability :
| Draws | |
|---|---|
So takes the values with probabilities . Its conditional mean is . Its conditional variance is
which matches and .
The bootstrap standard error is . The conventional plug-in standard error is . The two disagree by the factor .
Neither number is the population standard deviation. Without a model for how and were generated, nothing here tells us .
Connecting back to sampling risk
The bootstrap variance is computed from data. We can therefore ask how it behaves when the data are themselves random. Suppose are iid with mean and finite variance . Let denote the random version of .
Theorem 7.7.4
(Finite-sample downward factor for the mean)
Under iid sampling with finite variance ,
On average, the exact bootstrap variance of the sample mean understates the true sampling variance by the factor .
Proof
(Centering at the population mean)
Expand around :
This holds because the cross term is
which combines with .
Take expectations. Each , and . Hence
Dividing by gives , and dividing again by gives the claim. Comparing with yields the ratio .
Proof X-ray
Where the missing degree of freedom goes
The identity in the proof splits the spread of the data about the unknown into the spread about the sample mean and the term , which is the part absorbed by estimating . That term has expectation exactly , one observation's worth of variance, so the residual sum of squares has expectation rather than .
The resampling world never pays this price. Inside it the mean of is known exactly, so the bootstrap divides by , and inherits the deficit. Only uncorrelated observations and a finite variance are used; no normality is needed. Multiplying by turns into , so the same argument is also the familiar proof that is unbiased for .
Worked example 7.7.5
(Exact check of the downward factor at )
Here . Each deviation equals , so .
Independence gives , because the cross term vanishes. Hence and . The true variance is , and the ratio is , as claimed. For the dataset this is the factor between the two standard errors computed earlier.
Three cautions keep this result in proportion:
- It is a statement about expectations. On a particular dataset, may exceed or fall far below it.
- The relative deficit vanishes as grows, since . For small it is not negligible. At the bootstrap variance is 20% too small on average.
- It is specific to the sample mean. No general rule says bootstrap variances are conservative, or consistently anticonservative, for other statistics. For a median or a maximum the relationship must be studied separately, and it can behave quite differently.
The Monte Carlo layer
For the mean, has a closed form. For most statistics it does not, so we simulate. Conditional on the data, draw independent bootstrap samples and compute replicates . Let
Theorem 7.7.6
(Unbiasedness of the Monte Carlo variance estimator)
Conditional on the data, suppose are independent with common conditional mean and finite conditional variance . Then .
Proof
(The standard sum-of-squares argument, conditionally)
Apply the same identity as before, now under :
Each term in the first sum has conditional expectation . By conditional independence, . The right side therefore has conditional expectation . Dividing by completes the proof. Pairwise uncorrelatedness would in fact suffice.
The target here is , the conditional bootstrap variance. It is not the population sampling variance. Unbiasedness does not make exact: it is still a random number. With a finite conditional fourth moment its own conditional variance is ; for the sample mean here the empirical resampling space is finite, so this condition holds.
Common mistake
(Increasing to repair the bootstrap)
Finite adds Monte Carlo variability on top of whatever error the bootstrap idea itself carries. Increasing reduces the first error in the limit; it does not remove the second. For the dataset , approaches the exact bootstrap variance as , and the factor relative to the conventional remains. On an all-zero Bernoulli sample, as in the counterexample below, every gives .
The convention has conditional expectation . The difference from matters little for large , but reports should say which convention was used.
The proof assumed that the resamples are drawn independently given the data. That independence is created by the algorithm. It says nothing about whether the original observations were independent. That question belongs to the design, discussed below.
Bootstrap confidence intervals
Let be the conditional distribution function of the bootstrap statistic. Define its quantile by the left-continuous inverse
Definition 7.7.7
(Percentile and basic bootstrap intervals)
For a nominal level :
- the percentile interval is ;
- the basic interval is . It reflects the bootstrap quantiles about , treating as a stand-in for .
With simulation, is replaced by the empirical distribution of the replicates. One then needs an order-statistic convention, such as . Different software packages use different conventions.
The variance identities above give no coverage guarantee. Several factors determine actual coverage:
- the sample size;
- discreteness of , which may make the nominal level unattainable;
- bias and skewness of ;
- extreme-value statistics, for which naive resampling is known to behave poorly;
- whether the iid sampling model is correct.
Worked example 7.7.8
(Percentile intervals for the dataset )
Here has jumps of , , at , , , so , and .
At 90%. , since , and , since . The percentile interval is . Its conditional probability under is , not , because jumps from directly to at the value . With , the basic interval is also .
At 50%. and , so the percentile interval is , with conditional probability rather than . The attainable conditional probabilities are sums of adjacent jumps , , , so most nominal levels cannot be matched exactly.
When the data look falsely certain
Counterexample mode
An unseen rare value
Let be iid Bernoulli with fixed . The true variance of the sample mean is .
The event that every observation equals has probability . On that event:
- is a point mass at ;
- every bootstrap sample consists entirely of zeros;
- with conditional probability ;
- , and any equals ;
- both percentile and basic intervals collapse to .
The procedure reports complete certainty about a quantity that genuinely varies from sample to sample.
The bootstrap cannot resample a value it never observed. It explores only the support of the data, so it is blind to unobserved tail or rare-category behavior.
The counterexample shows that some datasets look falsely certain. It does not show that the bootstrap fails on every sample, nor that its large-sample theory is invalid. For the specified Bernoulli population, the event becomes rare as grows. Equally, zero sample variance does not prove that the population is constant. It shows only that this sample contained no variation.
What the bootstrap does and does not target
The nonparametric bootstrap computes features of the resampling law conditional on the observed empirical distribution. Treating these as approximations to the true sampling law of is an additional modeling step. That step relies on the observations being an iid sample, so that is a sensible stand-in for the unknown , and on being a statistic for which this plug-in works.
The resampling scheme must mirror the actual data-generating structure:
- Clustered data. Several rows per patient, school, or user may require resampling independent clusters instead of individual rows. Row-level resampling breaks within-cluster dependence and can misrepresent variability; validity depends on the sampling design.
- Time series. Serially dependent data need schemes that preserve dependence, such as block resampling.
- Whole pipelines. When evaluating a complete procedure, every data-dependent step must be repeated inside each resample. This includes standardization, imputation, feature selection, hyperparameter tuning, and model selection. Fitting these once on the full data and then bootstrapping only the final step can misrepresent variability and can leak information.
- Held-out test sets. A test set reserved for final assessment stays untouched throughout. It is never resampled into training, and it is never used for tuning.
Every such statement is specific to the target quantity and the resampling design. The bootstrap supplies none of the independence it requires.
Exercises
Exercise 7.7.9
(For the data , find the bootstrap distribution of , its conditional variance, and compare with and .)
Foundation
Solution(3)
- 1
The bootstrap distribution
Here , the deviations are , , and . The four equally likely ordered draws give , so takes the values with probabilities .
- 2
Conditional variance
This is the example shifted by , and variance is shift-invariant.
- 3
Compare
The single-draw bootstrap variance is , which is half of . The bootstrap variance of the mean, , is half of the conventional estimate . Both halvings reflect the factor .
Exercise 7.7.10
(For the data , compute , , the bootstrap variance and standard error of the mean, and . Give the full distribution of .)
Core
Solution(4)
- 1
Summary quantities
The mean is . The deviations are , with squares summing to , so and . The bootstrap variance is , and the standard error is .
- 2
Probability of an all-zero resample
Each draw equals with probability . Hence .
- 3
Full distribution
Key step
The bootstrap mean is . Therefore
Because: Only the number of draws equal to matters, and .
- 4
Check
and , in agreement with the theorem.
Exercise 7.7.11
(Three bootstrap replicate means are . Compute , and explain why it need not equal the exact conditional variance.)
Core
Solution(3)
- 1
Compute
The replicate average is . The squared deviations sum to , so . The convention would give .
- 2
Two datasets, one set of replicates
These three replicates are possible under the dataset , whose exact conditional variance is . They are also possible under , whose exact conditional variance is ; there the replicates take the values in some order with probability .
- 3
Interpretation
In both cases differs from the truth. The value is one random realization of an unbiased estimator, not the target itself; unbiasedness holds on average over repeated Monte Carlo runs. Agreement can also occur by chance at finite ; convergence as grows is a different statement.
Exercise 7.7.12
(Suppose the data are iid Bernoulli with . Find the probability that the exact conditional bootstrap variance of the mean is zero, the true standard error, and the smallest for which this probability is at most .)
Synthesis
Solution(4)
- 1
When the exact bootstrap variance vanishes
The exact conditional bootstrap variance is zero exactly when all observations are equal. Both all-zeros and all-ones produce it, so the probability is . A finite- Monte Carlo estimate can also be zero by chance, when all simulated replicate values agree, even for nonconstant data.
- 2
True standard error
The true standard error is . On either constant-data event the bootstrap standard error is ; the percentile interval is for all zeros and for all ones.
- 3
Smallest sufficient
We need . This sum decreases with positive integer ; at it is approximately , while at it is approximately . The smallest such is therefore .
- 4
Interpretation
Even then, the event is possible. The correct reading of an all-zero sample is that it is consistent with a small but positive success rate. Zero variability is not a finding.
Summary
Resampling from produces exact, data-only formulas: and . Under iid sampling this variance runs low by on average. Simulation with replicates adds a separate, reducible Monte Carlo error.
Intervals built from the bootstrap distribution carry no automatic coverage guarantee. The method can only reflect what the sample shows and what the resampling design respects.