Motivation
A table of observations is a finite mathematical object. A statement about what happens on the next observation requires something more: a model connecting the observed table to an unobserved population. This distinction explains why an exact least-squares calculation can coexist with uncertain predictions. Solving the normal equations removes an optimization problem; it does not remove the uncertainty created by observing only a sample.
Suppose a sensor records whether a measurement exceeds a threshold. The fraction of recorded exceedances is directly computable. The probability of an exceedance on a new trial is not another entry in the table. It is a parameter of a probability distribution. A useful analysis therefore says which trials are regarded as random, which distribution they share, and which kinds of dependence are excluded. Without these declarations, a familiar formula for a standard error can give a completely false impression of precision.
This note develops a deliberately small foundation. We use expectations, variances, independence, conditional means, and elementary inequalities to study sample averages and prediction loss. The results support Monte Carlo integration and held-out evaluation without asserting that every real data set satisfies the assumptions. Basic finite sums, integrals, and inner products are sufficient prerequisites. Conditional expectations below are understood in their usual almost-sure sense; no claim depends on choosing values at inputs of probability zero.
Definitions
A probability space consists of outcomes, measurable events, and a probability measure . A real random variable is a measurable function of the outcome. Its expectation is an integral with respect to , when integrable. A finite second moment means ; then its mean and variance are finite. The identity follows by expanding the square and using linearity. Finiteness of the variance is an assumption about the law; it cannot be inferred from the fact that a computer returns a number.
Definition 7.1.1
(Independent and identically distributed observations)
Random variables are independent when probabilities of intersections of events involving separate coordinates factor into products. They are identically distributed when each has the same probability law. We write i.i.d. when both conditions hold.
These are separate conditions. Independent measurements with increasing variances need not be identically distributed; identical copies of one random measurement need not be independent.
Definition 7.1.2
(Sample mean and sample variance)
For observations , the sample mean and, for , the sample variance are
The empirical mean is a random variable before the experiment and a number after the observations arrive. Its target remains fixed in this probability model. The variance describes individual observations; the variance of describes fluctuations of the estimator across repeated samples. Confusing them loses the entire benefit of averaging.
Definition 7.1.3
(Prediction risk, empirical risk and conditional mean)
For a pair with , a deterministic measurable predictor has squared prediction risk , assumed finite when used. Its empirical risk on observations is the average of the same loss over those observations. The conditional mean is the square-integrable function characterized by averaging given the information in ; its values are identified only up to sets of input probability zero.
An estimator may itself depend on training data. Calling such an estimator “fixed” in a test calculation means conditioning on the entire training procedure, including model selection and preprocessing, and then using test observations independent of that procedure. If test labels were used to choose the model, this conditional independence argument is unavailable.
Theorem/proposition
Theorem 7.1.4
(Mean-square error of an independent sample average)
Let be i.i.d. with mean and finite variance . Then
For , . The probability bound may exceed one, in which case the bound one is stronger. The result requires no normal distribution assumption.
The second result concerns a single new observation rather than an average.
Theorem 7.1.5
(Conditional mean and irreducible squared error)
Let and let be square integrable. With ,
Consequently minimizes squared population risk. Equality with its risk holds precisely when almost surely. This is a statement about the population distribution, not an algorithm for recovering from a finite sample.
Concept lensGeometric
The risk identity as Pythagoras in a space of random variables
Square-integrable random variables form an inner-product space with , and . The square-integrable functions of form a closed subspace, and is the orthogonal projection of onto it: the residual satisfies for every square-integrable . Since lies in the subspace, the risk identity is the Pythagorean theorem. Minimizing risk over functions of is therefore a best-approximation problem, exactly as least squares is in a finite-dimensional column space.
For a fixed predictor whose loss has finite variance , the first theorem applied to independent test losses gives an unbiased empirical risk with variance . Notice that alone does not guarantee : the latter concerns the second moment of a squared loss and may require fourth-moment control.
Proof sketch or proof idea
Proof
(Mean, variance, Chebyshev bound and the unbiased sample variance)
Linearity gives the expectation of the mean without independence. For its variance, expand the centered square into diagonal terms and cross terms. Independence gives when , leaving .
For a nonnegative random variable , the pointwise inequality implies for . Apply this to and .
The deterministic identity
has expectation . Dividing by proves unbiasedness.
Proof X-ray
What each assumption buys in the averaging proof
Unbiasedness of uses linearity only, so it survives any dependence. Independence enters once, to kill the cross terms in the variance; pairwise zero covariance would suffice for this identity, and full independence is a convenient stronger assumption. Identical distribution makes all diagonal terms equal to , so their sum is .
The probability bound comes from the Markov step applied to the squared error, which is why the variance must be finite. It concerns the distribution of the estimator over repeated samples; it is not a guaranteed error for the one sample observed.
In the sample-variance identity the subtracted term has expectation by the variance computation. This is the single unit of variance lost by centering at instead of , and it explains the divisor . Dividing by would underestimate the variance on average. Unbiasedness of does not imply unbiasedness of ; a nonlinear function does not commute with expectation.
The risk identity uses a different ingredient, the defining property of the conditional mean.
Proof
(Expanding the risk around the conditional mean)
Write and expand. The cross term vanishes because and is a function of . Square integrability justifies the products by Cauchy–Schwarz. The remaining difference in risks is the expectation of a nonnegative square, which is zero exactly when the square is zero almost surely. This is the probabilistic version of an orthogonal projection argument.
Worked examples
Worked example 7.1.6
(Threshold counts and a conservative sample size)
Let indicate a threshold exceedance, with independent trials and common probability . Since , its variance is . To ensure uniformly over , the theorem permits . This is a sufficient conservative sample size, not a claim that 1999 observations necessarily fail. For and , the exact variance is and the standard deviation of the fraction is . Neither number is the variance of one indicator.
Worked example 7.1.7
(Integration by uniform sampling and a control variate)
Take independent uniform on . Then and , so the average estimates the integral of with variance . Its root mean-square error is . Multiplying the number of samples by four halves this error; multiplying by two does not. This rate is independent of the nominal dimension for an integrand of controlled variance, but the variance itself can change greatly with dimension and with the sampling distribution.
An alternative unbiased estimator is , since . Its variance is : expand the fourth-degree polynomial and integrate each monomial. The standard deviation is four times smaller than for .
The coefficient is not lucky. Subtracting preserves the expectation for every real , and the variance is a quadratic in with minimum at , because . Choosing a coefficient from the same finite sample can change unbiasedness and requires another analysis.
Worked example 7.1.8
(Prediction has two different error sources)
Let be uniform on , and write , with independent noise satisfying and . The conditional mean is . For the predictor , the risk is . For , it is . Learning the correct slope removes the approximation error but cannot remove the noise variance. If the slope was estimated from a separate training sample, condition on that estimate before using this calculation; average over its training randomness only afterwards.
The averaging theorem assumed independence. Equal marginal laws are not a substitute, as the next check shows.
Counterexample mode
Equal marginals do not make independent observations
A useful dependence counterexample uses a single Bernoulli variable and sets for every . Each coordinate has the same mean and variance as an independent Bernoulli observation, but for every sample size. The mean variance remains . More generally, equal means and finite second moments give . A long correlated record can contain far less information than its row count suggests.
There is also a useful distinction between random design and fixed design. In the sensor example, treating all measurements as independent draws models variation in the sampled situations as well as measurement noise. In a laboratory experiment the input locations may instead be selected in advance. Then a conditional analysis averages only over the measurement noise at those locations. Both models can be appropriate, but they answer different questions about repetition. A standard error computed for one model should not be transferred silently to the other.
Common mistakes
Common mistake
(Reading a probability guarantee as a property of the sample in hand)
An empirical average does not equal its population target by definition. Unbiasedness is a statement about its expectation, not about each realization. A high probability guarantee does not certify that the particular sample you hold is in the favorable event. Replacing the unknown variance in a probability inequality by a random sample variance requires a new theorem, for example confidence-interval theory under specific distributional assumptions; the inequality proved above does not authorize that substitution.
Common mistake
(Splitting records instead of sampling units)
Independence must apply to the unit of sampling. Several records from the same subject may share random effects. Randomly splitting those records can create dependence between training and testing even when the code performs a random shuffle. A split by subject may be needed to match the intended prediction question.
Common mistake
(Averaging without a finite variance or the right law)
Finite variance is substantive. A standard Cauchy random variable has no integrable mean or finite variance, so the theorem cannot be invoked merely because a computer produced a numerical average. Likewise, Monte Carlo error describes random integration error under the stated law; it does not include bias from a simulator that samples the wrong law.
Summary
An average estimates an expectation only after a probability model identifies that expectation. Under independent equal-distribution sampling with finite variance, its mean-square error is . Squared prediction risk separates conditional noise from mismatch to the conditional mean. Independent test observations let us apply the averaging result to a fixed trained rule. These results establish useful foundations while leaving concentration inequalities with sharper rates, dependent sampling, and general learning guarantees for later study.
Exercises
Quick check
(Independent measurements have common mean and variance nine. How many observations suffice, using the proved inequality, to make the probability of absolute mean error at least at most ? State what “suffice” means.)
Solution · Answer
The bound is . Requiring this to be at most gives . Under the stated sampling model every such sample size meets the probability bound. This is a sufficient condition from a possibly loose inequality, not a necessary sample size and not a deterministic error guarantee.
Quick check
(Let be uniform on . Find the expectation and variance of , and the root mean-square error of the average of independent copies. What sample-size factor reduces that error by three?)
Solution · Answer
Integrating monomials gives and . Thus . The average is unbiased with root mean-square error . Increasing by a factor nine divides this error by three. No claim about a fixed realization follows from the root mean-square calculation alone.
Quick check
(Let take values minus one and plus one with equal probabilities, and let , where is independent with mean zero and variance two. Find the risk of every affine predictor and identify all minimizing pairs.)
Solution · Answer
The conditional mean is . Expanding its squared difference from and using , gives . The unique minimizing pair is , , with risk two. The uniqueness here concerns these affine parameters; population predictors in general are identified only on the support of .
Quick check
(Suppose random variables have common positive variance and every distinct pair has covariance . Derive the variance of their average. Explain the obstruction to variance reduction when a fixed positive persists as grows.)
Solution · Answer
There are diagonal covariance terms and off-diagonal terms. Therefore the variance is . For fixed positive it tends to , so collecting more correlated copies does not remove the common randomness. A proposed covariance structure must also be positive semidefinite; for a fixed with , the equicorrelation parameter lies between and one.