What varies when a model is trained again?
A model can fit one sample well and still make unreliable predictions for new data. To study that difference, imagine repeating the entire training procedure on independently drawn datasets. At a fixed input , its prediction then varies with the training data. We will decompose average prediction error into three terms, but the result depends on the loss function and on what is independent of what.
Let be a random training dataset and let be the resulting real-valued prediction at a fixed . For a fresh response at that input, write , where , , and . Assume the test response noise is independent of at this fixed input and the relevant second moments are finite. Set . Expectations over describe hypothetical repeated training samples, not variation of predictions within a single fitted model.
Definition 10.1.1
(Bias and variance of a procedure at a point)
For the training procedure that produces , its bias at is , the gap between the average fitted prediction and the conditional mean. Its prediction variance at is , the spread of the fitted prediction around across training sets. Both are properties of the procedure together with the data-generating distribution, not of one fitted model.
The decomposition
Theorem 10.1.2
(Pointwise bias–variance decomposition)
Under the conditions above (a fresh response independent of at the fixed input , finite second moments), the pointwise expected squared prediction error is
The three terms are fresh-response noise, squared bias and prediction variance. The variance is around the average fitted prediction , not around the true response. An unbiased procedure has ; it need not have low variance. The noise term remains even if is known exactly, when predicting a new noisy response rather than its conditional mean.
Proof
(Expanding the square around the average prediction)
Write
Here is a constant, by the definition of , and . Expanding the square gives
The terms with as a factor vanish because . Since is independent of and is a function of , . What remains is , and .
Proof X-ray
Where independence enters, and what the middle term does
The key move is to insert , so that the prediction error splits into a noise part, a constant part and a centered random part. Centering is what makes two of the three cross terms vanish for free, since a constant times a mean-zero quantity has mean zero. Independence of the fresh noise from is used exactly once, for . If the response being predicted shared noise with the training data, this term would be nonzero and the clean three-term identity would fail. Finite second moments are needed for every expectation above to exist.
The identity can be averaged over random test inputs that are independent of the whole training dataset when the stated fixed-input conditions hold and the relevant expectations are finite. If test inputs and training data are dependent, use the training distribution conditional on the test input and recheck the noise assumptions. The identity does not imply that the three quantities can be recovered from one dataset without additional assumptions.
Concept lensGeometric
The decomposition as the Pythagorean theorem
In the space of square-integrable random variables with inner product , the three pieces , and are pairwise orthogonal: has mean zero and is independent of , is constant, and has mean zero. Hence , which is the decomposition. Orthogonality, not any distributional shape, is all the proof uses.
A worked shrinkage family
Worked example 10.1.3
(Shrinking a sample mean toward zero)
Suppose independent observations each have mean and variance , and the training set contains observations. Let be their mean. Compare estimators for a fixed, prespecified . The sample mean has expectation and variance , so
| Fixed multiplier | Squared bias | Variance | Mean squared estimation error |
|---|---|---|---|
| 1 | 0 | 3 | 3 |
| 1/2 | 1 | 3/4 | 7/4 |
| 0 | 4 | 0 | 4 |
The half-shrunken estimator has smaller error here, despite being biased. But performs worse than no shrinkage for this mean; at , its estimation error would instead be zero. Thus shrinkage is not uniformly best merely because it reduces variance.
Differentiating gives the oracle minimizer , with error . This is an oracle calculation: the general optimum uses the unknown , and cannot be chosen from that formula as if the true mean were known. If a fresh observation from the same population is predicted by , its expected squared prediction error is the estimation error plus , because that fresh observation is independent of the training sample.
Counterexample mode
Shrinkage does not always reduce error
Claim: since shrinking multiplies the variance by , the estimator with always beats . Take , variance and as above, but with true mean . Then has error , while has squared bias and variance , so error . Even at the extreme choice gives error . The correct statement: for fixed and sample-mean variance , has smaller error than exactly when . For and this is , which holds at and fails at .
The same family describes a regularized objective. For a fixed , minimize . Differentiation gives , a multiplier in for finite ; is approached only as . The penalty here shrinks toward zero; a different target or an unpenalized intercept changes the calculation. The example is an intercept-only illustration, not a claim that every ridge coefficient has this same scalar variance.
Worked example 10.1.4
(A penalty as a multiplier, with its two limits)
With and , the multiplier is , so the error is , smaller than . With the multiplier is and the error is , matching the table. At the limit the multiplier is and the error is ; as the multiplier tends to and the error tends to . The error is therefore not monotone in : it falls below for moderate and rises above again once is large (for instance gives and error ; the error exceeds exactly when ).
Worked example 10.1.5
(Two procedures at one input)
Suppose at a fixed the conditional mean is and the noise variance is . Procedure 1 has and prediction variance ; procedure 2 has and prediction variance . Procedure 1 is unbiased but its expected squared prediction error is . Procedure 2 is biased, with squared bias , and its error is . Even a procedure with and variance cannot go below for a fresh noisy response, so the noise term is an edge case that no choice of procedure removes.
Choosing and evaluating a procedure
In practice, a multiplier or penalty is usually selected from data. Then it is itself random, and substituting a data-chosen into the fixed- variance formula is generally invalid. Evaluate the entire selection procedure, including preprocessing, using training-only steps within each validation fold. Keep an independent test set for final assessment, rather than tuning repeatedly on it. Cross-validation estimates a procedure's performance under its split and data assumptions; it does not prove an exact noise, bias or variance decomposition for one fitted model.
Common mistake
(Using a data-chosen multiplier in the fixed-multiplier formula)
The formula is for a multiplier fixed before seeing the data. If one replaces the unknown in by , the multiplier is itself random and depends on the same sample. The variance and bias of are then no longer and , and must be recomputed for the whole procedure.
Common mistake
(Evaluating a fit on the observations it was trained on)
Predict each training observation by the sample mean with and variance . Then has variance , so the expected squared error on a training point is . For a fresh independent response it is . The training value is smaller because appears on both sides, so the cross term is nonzero.
The decomposition does not say that training error equals test error, that bias and variance always move in opposite directions, or that squared loss is suitable for every task. Also, independence of a fresh test response is essential to this proof; evaluating on the very observations used for training can retain nonzero cross terms.
Exercises
Quick check
(With a fixed true mean and sample-mean variance , derive the risk of for fixed real and find its minimizer. What happens when ?)
Add squared bias and variance, then differentiate in .
Solution · Guided solution
The bias is and the variance is , so the risk is . Its derivative vanishes at , and the risk is convex in , so this is the minimizer. For the minimizer is and the minimum risk is : the best multiplier discards the data entirely, which is correct only because the true mean is exactly zero.
Quick check
(In the worked family, compute the expected squared error for predicting an independent fresh response when . Explain why this differs from estimation error.)
Add the noise variance of the fresh response to the estimation error.
Solution · Answer
The estimation error is and the noise variance is , so the expected squared prediction error is . It differs because a fresh response contains its own noise, which no estimator of the mean can predict.
Quick check
(Explain why fitting a scaling transform once on all data before validation folds are formed can invalidate the intended separation of fitting and validation.)
Ask which observations influenced the fitted scale.
Solution · Answer
The fitted scale then depends on the validation observations, so each validation fold has influenced the model it is used to assess. The validation error is no longer an error on data independent of fitting, and tends to be optimistic. The scaling must be fitted on the training part of each fold only.