Motivation
A fitted model has already seen the noise in its training responses. Training error therefore measures an easier task than predicting an independent repeat of those responses. Complexity corrections try to account for this advantage, but different criteria target different quantities. A correction for squared prediction error is not automatically a correction for log density, and a Bayesian integrated likelihood is not the same target as either.
We begin with a finite calculation for a fixed linear fitting rule. It identifies exactly how much training error is optimistic, even when the fitted mean model is biased. We then derive a finite Gaussian correction when both regression coefficients and noise variance are estimated. The inverse residual variance is the source of an additional small-sample term.
Only after those exact results do we discuss general information criteria. Their compact formulas hide assumptions about parameter dimension, likelihood regularity, sampling and the comparison target. The aim is to know when an elementary proof applies, when an asymptotic statement is being invoked, and when neither provides a justified conclusion.
Definitions
Definition 26.1.1
(Fixed-design prediction and a linear smoother)
Let , with deterministic and . Let be an independent repeat at the same design points. For a deterministic matrix , fit . Training squared error is ; repeat-response error is . Define effective degrees of freedom .
The matrix must be fixed independently of these responses. A ridge matrix at a predetermined penalty is an example. A matrix chosen after searching the observed responses is not covered simply by inserting its realized trace. Random designs can be handled by conditioning on the design when the stated noise assumptions hold conditionally.
Definition 26.1.2
(Likelihood scores and parameter counting)
For a model fitted by maximum likelihood, let be its maximized deviance and its number of independently fitted real parameters. Conventional scores are and . Count an estimated intercept and an estimated common variance, but not quantities fixed in advance. In full-rank Gaussian regression with mean coefficients and unknown variance, .
Comparisons require the same observations and the same underlying response measure. Constants common to every model can be discarded; constants that change with response transformations or data subsets cannot. Maximized unpenalized likelihood and a penalized objective are also different quantities. Effective dimension for a fixed smoother describes its response sensitivity; it should not be silently substituted for a regular likelihood parameter count in every information criterion.
Theorem/Proposition
Theorem 26.1.3
(Finite prediction optimism and the Cp correction)
For the fixed smoother above,
Thus is unbiased for average repeat-response error, and subtracting makes it unbiased for . If is an orthogonal projection of rank , the normalized quantity estimates mean-vector prediction error divided by .
Theorem 26.1.4
(An exact Gaussian small-sample deviance correction)
Suppose , with deterministic full-column-rank , correct Gaussian errors, unknown , and . Fit by least squares and by maximum likelihood. For an independent repeat at the same design, let . Then
With , an unbiased corrected score for this expected deviance is
This exact statement assumes a fixed, correctly specified regression model; it does not assert exactness after model search, under misspecification or for arbitrary nonlinear models.
Theorem 26.1.5
(A sufficient Laplace condition for the BIC leading term)
Consider a fixed-dimensional -parameter model with likelihood , an interior maximizer , and prior density continuous and positive at . Extend the rescaled integrand by zero outside its parameter domain. Suppose
pointwise, with , and suppose an integrable envelope independent of bounds these integrands. Then marginal likelihood satisfies . For random samples the same conclusion is conditional on a setting where these sufficient conditions hold, or requires their probabilistic counterparts.
Proof sketch or proof idea
Proof
(Expand the two squared errors before canceling)
Set . Independence and zero means give
whereas
The bias and fitted-noise variance cancel, leaving the optimism. Removing the independent future-noise contribution proves the mean-vector version. For a projection, . Correct specification of the mean was not used, so this finite squared-error calculation permits approximation bias. In fact only the stated second moments and independence are needed for this proof; Gaussianity is needed for the next exact deviance calculation.
Proof
(Independent Gaussian subspaces produce AICc)
Let and . Orthogonal coordinates diagonalize into ones and zeros. Isotropic Gaussian coordinates are independent, so and are independent. Consequently is independent of and of .
The Gaussian likelihood gives and . The expected numerator is , since independent future noise contributes and fitted-mean noise contributes .
The chi-square density is proportional to . Integrating after dividing by and using the gamma recurrence gives for . Therefore
Writing this correction as proves the formula. If , the inverse moment diverges; a negative or zero denominator is not an invitation to use the algebraic expression anyway. Replacing the MLE divisor by an unbiased variance divisor changes the deviance calculation and must be rederived.
Proof X-ray
Two exact corrections have different assumptions
The Cp argument allows a biased fixed linear mean fit with known variance because its target is squared error. The exact AICc argument additionally uses a correctly specified mean, Gaussian independence of projected and residual components, and a finite inverse chi-square moment. Neither proof remains unchanged when fitting, variable selection or penalty tuning makes the rule data dependent.
Proof
(The dimension penalty comes from a volume factor)
In the marginal likelihood integral, set . The Jacobian contributes . Under the theorem's pointwise and envelope assumptions, dominated convergence gives
Taking minus twice the logarithm proves the BIC leading expression. A local Hessian calculation alone would not justify integrating over the entire parameter space: the envelope or a separate tail-concentration argument is essential. The bounded remainder includes prior density and curvature terms, which can differ between models. BIC is therefore a leading approximation, not an exact finite-sample posterior probability.
General AIC uses a different asymptotic argument: for a correctly specified, identifiable, fixed-dimensional smooth likelihood with an interior true parameter, nonsingular Fisher information and sufficient moment/tail control for expected likelihood expansions, fitted-parameter optimism has leading penalty . That general asymptotic result is external here; we proved its finite Gaussian relatives rather than asserting a universal theorem. Under misspecification the curvature and score covariance need not coincide, so the same parameter-count penalty is not automatically justified. Singular mixtures and latent models can also violate the regular Laplace conditions behind ordinary BIC.
Parameter counting and effective dimension should be checked from the actual fitting rule. For a fixed ridge rule , an SVD gives , a continuous measure rather than the number of nonzero coefficients. If an intercept is fitted without penalization and the penalized predictors are centered, the orthogonal constant direction contributes one. If is selected using , the fixed-rule proof does not correct the extra search. Estimating also introduces uncertainty; simply substituting an arbitrary residual variance does not retain the exact known-variance unbiasedness claim.
Before computing a score, write down the response vector, fitting rule, prediction target and all estimated parameters. Then identify which correction has actually been justified. An exact unbiased score for each fixed candidate does not make the minimum over many candidates an unbiased estimate of the winning procedure's performance. This is the same selection effect developed in the sparse-learning unit. Reporting an independent evaluation of the entire search preserves the distinction between a useful selection score and evidence about the selected model.
Worked examples
Worked example 26.1.6
(Training and repeat error for a projection)
Take , projection rank three, variance two and a correctly represented mean. Expected training error is , while expected repeat-response error is . The correction is . Subtracting future noise twenty leaves mean-estimation error six. If the mean has approximation bias of squared norm five, both training and repeat expectations increase by five, and their difference remains twelve.
Mean-vector error measures how well the fitted vector estimates the noiseless means. Repeat-response error adds fresh observation noise. Training error instead reuses the noise that shaped the fitted vector, creating an advantage rather than merely adding fresh noise. The two corrections in this example therefore have different signs and purposes; subtracting future noise is not the same operation as correcting training optimism.
Concept lensAnalytic
Optimism as covariance with one's own observation
For the fixed smoother, , so the optimism equals : twice the total covariance between each fitted value and the response it was fitted to. Effective dimension thus measures how strongly each fit follows its own observation, which is why a ridge fit can cost less than the number of its coefficients.
Worked example 26.1.7
(A fixed ridge rule has fractional dimension)
Suppose the two nonzero squared singular values of are nine and one, with . Ridge smoothing eigenvalues are and , so effective dimension is . For variance one, squared-error optimism is . Counting two fitted coefficients would instead suggest four. The fitted-noise variance is , which is different again. These traces answer different questions even though they come from the same smoother.
Worked example 26.1.8
(A finite correction can be substantial)
Let and fit mean coefficients plus unknown variance, so . AIC adds eight to the maximized deviance. The exact Gaussian AICc correction adds , approximately 10.67. BIC adds , approximately 11.98. Their numerical ordering does not make them estimates of one common quantity: the first two concern predictive deviance under their respective assumptions, while BIC approximates integrated-likelihood comparison.
Common mistakes
Counterexample mode
The nominal parameter count can miss adaptive search
It is tempting to believe that, even when the projection is chosen from the data, the optimism is still times the trace of the chosen matrix. It is not. Take , , and the two fixed coordinate projections and . Choose the one with larger projected energy , so with . The chosen matrix always has trace one, but the training error is , and the expected repeat-response error is . Since and , where and are independent standard normal variables, we get . The optimism is therefore , not . The independence behind the fixed-matrix cancellation fails once the matrix depends on the response; independent evaluation of the complete search procedure remains a valid way to assess its prediction performance.
Common mistake
(AICc is not a universal repair for every small sample)
The exact correction proved here is for a fixed correctly specified Gaussian regression with enough residual degrees of freedom. Substituting a mixture's informal parameter count into the same rational expression does not repair covariance collapse or nonidentifiability. Similarly, a small BIC difference is not itself a calibrated posterior probability unless the omitted model-specific terms and prior model probabilities have been handled.
Common mistake
(Two different traces, one smoother)
For a fixed smoother, sets the optimism, while enters the variance of the fitted values, and the expected noise part of the training error is . For a projection the two traces coincide, which hides the distinction. For the ridge smoother above, with eigenvalues and , the traces are and , so the noise part of the training error is . Using in its place would be wrong.
History of an idea
Distinct motivations, explicitly checked sources
Akaike's 1974 original article formulates a likelihood criterion with a fitted-parameter penalty. Schwarz's 1978 original record describes a Bayesian asymptotic model-dimension argument. Hurvich and Tsai's 1989 paper develops a small-sample correction; the finite regression identity above is proved with its own stated target and counting convention. Section 2.1 of Flynn, Hurvich and Simonoff's primary paper on penalized likelihood under misspecification explicitly lists the normalized AICc formula; that broader study is not being cited as an original derivation of every result here.
Summary
For a fixed linear fit, covariance algebra identifies an exact squared-error optimism correction. Correct Gaussian regression with estimated variance supplies an additional inverse-variance correction and the stated AICc formula. AIC's general likelihood interpretation and BIC's marginal-likelihood expansion require further regularity. Parameter count, smoother trace, residual degrees of freedom and search complexity should never be treated as interchangeable names for one number.
Exercises
Quick check
(A fixed rank-four projection uses twelve observations with variance three. Its squared mean approximation bias is two. Find expected training error, repeat error and mean-vector prediction error.)
The bias appears in all three quantities.
Solution · Answer
Expected training error is . Repeat-response error is , giving optimism 24. Mean-vector prediction error is , obtained by removing future-noise contribution 36 from repeat error. An unbiased observed mean-risk estimate would be training error plus 24 minus 36, not merely the residual sum itself.
Quick check
(A fixed ridge smoother has eigenvalues , and , with all other eigenvalues zero. Compute effective dimension, fitted-noise variance in units of , and squared-error optimism.)
Distinguish the trace from the trace of the square.
Solution · Answer
Effective dimension is . The fitted-noise variance multiplier is . Optimism is . These calculations assume the smoother is fixed before observing the responses; selecting its eigenvalues from response noise would require another analysis.
Quick check
(In correct Gaussian regression with n equal ten and p equal four, compute the exact deviance correction and compare it with AIC. Explain what fails if n is reduced to six.)
Count the variance parameter and examine the inverse chi-square moment.
Solution · Answer
There are parameters. The exact correction is , while AIC adds ten. At , residual degrees of freedom are two and the inverse chi-square mean diverges. The denominator becomes zero, reflecting an infinite expected repeat deviance in this formulation, not a finite correction that can be patched by omitting a term.
Quick check
(Explain why a local nonsingular Hessian is insufficient by itself to justify a BIC expansion. Identify the global condition used in the theorem and explain why ordinary BIC need not apply to singular mixtures.)
The marginal likelihood integrates outside the local neighborhood too.
Solution · Answer
A Hessian describes likelihood shape near one maximizer. Another region can contribute substantial mass, or tails can invalidate exchanging a limit with integration. The theorem imposes an integrable envelope on the complete rescaled integrand, making dominated convergence legitimate. Regular identifiable models may establish analogous control through concentration arguments. Singular mixtures can have nonisolated parameterizations, degenerate information or unbounded likelihood, defeating the ordinary quadratic and volume calculation.