A small singular value can make an unregularized least-squares coefficient react strongly to noise. Ridge regression changes the optimization problem so that every coefficient is penalized. This unit uses the unscaled squared-loss convention; adding a factor to the loss changes the numerical meaning of the penalty.
A unique solution even when columns are dependent
Definition 10.2.1
(Ridge objective)
Let , , and fix . Penalize all coefficients, with no separate unpenalized intercept:
The ridge estimator for a given is a minimizer of over .
Expanding and differentiating gives , and the question is whether the stationary point is the only minimizer.
Theorem 10.2.2
(Uniqueness of the ridge solution)
For every , every and every , the matrix is invertible and has exactly one global minimizer,
Proof
(Positive definiteness and an exact quadratic expansion)
For nonzero , . A positive definite matrix is invertible, so is defined and satisfies . For any , expand . The linear term in is , so
For this is strictly positive. Hence is a global minimizer and no other point attains the minimum.
Proof X-ray
Where the penalty is used
The expansion is exact because is a quadratic polynomial, so a stationary point is automatically a global minimizer; no convexity theorem is needed. The only place the penalty enters is the term , which is strictly positive for every . The data term alone is only nonnegative, and it vanishes along . Without the penalty () a rank-deficient would therefore give a whole affine family of minimizers, as the rank-deficient example below shows. The argument also fails if some coefficient direction is left unpenalized, because along that direction the second term disappears.
The inverse is valid even when has dependent columns or more columns than rows. This conclusion uses and a penalty on every coefficient; an unpenalized intercept or other unpenalized direction needs its own rank check.
What the singular values do
For rank , take a thin singular-value decomposition with and orthonormal and . The following result expresses the ridge fit through this decomposition.
Theorem 10.2.3
(Spectral form of the ridge fit)
With the thin singular-value decomposition above and ,
Proof
(Checking the normal equation on each singular direction)
Since and the are orthonormal, . Put . Then , so solves the normal equation and equals by uniqueness. Finally gives the formula for .
The formulas remain valid for , interpreting the empty sums as zero. Relative to the unregularized nonzero-singular-direction fit, the fitted response is multiplied by a filter between zero and one. For fixed , smaller receive a smaller filter. As the fit tends to zero. As it tends to the orthogonal projection onto the column space of ; when is rank deficient, this limiting statement does not make invertible at .
Concept lensGeometric
Ridge as a fit constrained to a ball
Put . Then also minimizes over the ball : a competitor in the ball with smaller loss would have . Larger penalties correspond to smaller balls, and the ridge point is where the level ellipsoids of the loss first touch the ball.
Worked example and a measure of fit flexibility
Worked example 10.2.4
(Ridge on a diagonal design)
Take , , and . The penalty and loss use the convention above. Then
The filters are and , so the direction with the smaller singular value is shrunk more. The residual sum of squares is .
Here the fitted-value operator is , with . More generally , where and
For fixed and fixed , suppose with and . Then . This motivates calling the trace effective degrees of freedom for this linear smoother. It need not be an integer. If is chosen using , the operator is no longer fixed as a function of the data, so this covariance identity cannot simply be reused with the selected value.
Counterexample mode
A larger penalty does not always lower coefficient error
Claim: a positive penalty can only help recover the true coefficients. Take and noise-free data with , so . At the least-squares solution is itself, with error . At the ridge solution is , with squared coefficient error . Shrinkage helps only when noise variance outweighs the bias it introduces; with exact data it only adds bias. A large penalty produces a small trace and often a simpler fit, but this alone does not prove low prediction error. Assess a data-chosen penalty with a procedure that keeps preprocessing and tuning inside the training folds; the fixed-penalty calculation above is not itself an out-of-sample guarantee.
Common mistake
(Comparing penalties across different loss scalings)
The objective has the same minimizer as . With , the value in the scaled convention corresponds to in the convention of this unit. Quoting a penalty value without the loss convention does not determine the amount of shrinkage.
Worked example 10.2.5
(A design with a zero singular value)
Let , and . Here , , , so and . Direct check: and , giving . The second response entry lies outside the column space of and is never fitted, at any , and the second coefficient is because it is penalized but does not affect the loss. The effective degrees of freedom are , not an integer.
A separate rank-deficient example: which coefficient is identified?
The diagonal example above has independent columns. To see what changes when two features cannot be distinguished by the training data, consider a different design and response:
Worked example 10.2.6
(Equal fits, different coefficients in a rank-deficient design)
We keep the same objective : no loss normalization, no separate intercept, and a penalty on both coefficients. Write . Then , so the data-dependent part of the objective is . The vector lies in : adding any multiple of it changes the two coefficients without changing either training prediction. The observations determine the sum relevant to their fitted values, not the individual coefficients.
First take . Every vector with fits both training targets exactly. In particular, and are distinct least-squares solutions. Their squared norms are and , respectively. The first is the minimum-norm solution, but that choice is an extra convention: both vectors give the same observed fit.
For , the normal equations become
Subtracting the equations gives , where . The general positive-definiteness argument above makes the unique ridge minimizer. To compare it with another vector giving the same training predictions, move along the null direction to . The residual is unchanged, while the squared coefficient norm increases by ; hence the penalized objective increases by exactly . Ridge chooses one point from this equal-fit line by its penalty. It does not add a new observation or change .
At , the anchor is and its training predictions are . With , the comparison vector is and has those same predictions. Both residual sums of squares equal , but their penalized objectives are and , a gap of . At a new feature vector their predictions are and . This contrast is outside the observed design direction; no target for it has been supplied. Different predictions there do not by themselves establish which vector predicts future outcomes better.
The regularized normal matrix has eigenvalues and . Its 2-norm condition number is therefore for —equal to at —and the matrix is singular at zero. This condition number is for , not for . The interactive comparison below uses precisely this separate example; varying the penalty, null-direction displacement and query lets you check which quantities change.
The comparison assumes the fixed design, equal feature scaling and the stated penalty convention. A rescaled loss or an unpenalized intercept changes the numerical problem. In particular, uniqueness of this penalized estimate alone does not prove that the data identify two independent feature effects or that the estimate has smaller test error.
Exercises
Quick check
(For , what are the coefficient and fitted response at ?)
Use the normal equation.
Solution · Answer
The normal equation is , so both the coefficient and the fitted response are zero.
Quick check
(In the diagonal worked example, compute .)
Use the residual vector .
Solution · Answer
.
Quick check
(If a positive singular value satisfies , what is its fitted-value filter?)
Substitute into .
Solution · Answer
.
Quick check
(Why does positive not by itself establish uniqueness if some coefficient direction is unpenalized?)
Take a design whose second column is zero and penalize only .
Solution · Answer
In the proof the term is what makes the difference strictly positive. If a direction is unpenalized and , the difference is along it. For instance, if the second column of is zero and only is penalized, does not depend on at all, so every value of is a minimizer.