Weak directions are not always zero directions
A least-squares problem asks for minimizing . A rank-deficient matrix can have many minimizers; even a full-rank matrix can have a direction that changes only slightly. Small changes in may then produce large changes in the fitted coefficients. Singular values separate these two phenomena and make the effect of regularization explicit.
Let have rank .
Definition 4.3.1
(Compact singular value decomposition)
A compact singular value decomposition of is a representation
where the and are orthonormal families. The span , the span , and .
These are exact-arithmetic statements; numerical rank depends on scale and data accuracy.
The minimum-norm least-squares solution
Definition 4.3.2
(Pseudoinverse and minimum-norm solution)
The pseudoinverse of is the operator . The minimum-norm least-squares solution for data is .
Theorem 4.3.3
(Minimum-norm least-squares solution)
Let . A vector minimizes if and only if it has the form with , where
In particular is the unique minimizer of smallest Euclidean norm. If , then , so a perturbation of the data changes by at most , and this bound is attained.
Proof
(Splitting the objective along the singular vectors)
Every can be written with , because the and span . Write and . Since and , orthogonality gives
Hence every least-squares minimizer has , and is arbitrary. Because the are orthogonal to , , which is smallest, and uniquely so, when . This gives . Since sends each to this unique vector, it does not depend on the chosen SVD. For the sum is empty and . For and , Bessel's inequality gives , with equality for . So , and a data perturbation changes by , of norm at most .
Proof X-ray
Where the separation comes from
The proof turns one minimization into independent scalar ones, and two orthogonality facts do all the work. The are orthonormal and , so the cross terms vanish and the objective splits into plus a constant. The are orthogonal to , so splits in the same way. Positivity of is used exactly once, to solve ; a zero singular value would leave unconstrained, which is precisely a kernel direction. The residual sees only , so every kernel vector leaves it unchanged. That is why minimizers differ, and why choosing gives the one of smallest norm.
A small positive singular value is not a null direction, but it can amplify noise greatly.
Ridge filtering
Definition 4.3.4
(Ridge regularization)
For , ridge regularization chooses the unique minimizer of
Theorem 4.3.5
(Ridge solution and its noise bound)
For the objective above has a unique minimizer , given by
Equivalently, is the unique solution of . For every , , so changing by changes by at most . For fixed , as .
Proof
(Scalar quadratic problems along the singular vectors)
The decomposition above reduces this to independent scalar quadratic problems. Their minima give
The kernel component is zero because . Equivalently, solves ; the matrix is positive definite even when is rank deficient. The scalar filter is bounded uniformly in :
because . Thus changing by changes by at most . For fixed , as . For , the actual amplification factor is strictly below , although the looser uniform bound need not be. The solution is generally biased relative to .
Worked example 4.3.6
(Ridge halves a spurious component)
Let , and suppose the exact right-hand side is while the observed data are . Unregularized inversion gives , even though the exact solution for the exact data is . With , ridge gives
It halves the spurious second component in this example, while changing the first component slightly. The example illustrates a tradeoff, not a universal choice of .
Worked example 4.3.7
(Ridge on a rank-deficient matrix)
Let and , so , and . Every minimizer has first coordinate and arbitrary second coordinate , and the residual has norm ; the minimum-norm solution is . For , the filter gives . Check: and , so solves the regularized system. The kernel direction receives coefficient zero for every , and the first coordinate is shrunk from to , which is the bias of ridge.
Truncated singular values
Definition 4.3.8
(Truncated SVD solution)
For a threshold , define
This truncated SVD discards directions below the threshold. If the data change by , orthogonality gives . But a discarded component may contain real signal, so the same threshold that limits noise amplification can introduce approximation error. Precisely, .
Counterexample mode
Regularization can remove a genuine signal
Keep , but take exact noiseless data . The true minimum-norm solution is . Ridge with returns ; truncated SVD with returns zero. Neither choice is universally better without assumptions about noise and the desired signal.
A practical threshold or ridge parameter should reflect data accuracy, scaling and the purpose of the solution. A larger residual alone does not prove that a regularized estimate is worse for the unknown true signal; conversely, a smaller residual does not establish that noisy coefficients are accurate. Computing an SVD or solving a regularized system in floating point introduces additional errors not analyzed by these exact formulas.
Concept lensAnalytic
One filter formula for three solutions
Each solution above has the form
with a filter applied to the coefficients of . The pseudoinverse uses . Ridge uses the smooth filter , which equals at and tends to for . Truncation uses for and otherwise. Seen this way the two regularizers differ only in how sharply they switch off weak directions. In the two-direction calculation, the filter value at and is , which is the halving observed there.
Worked example 4.3.9
(Truncation and ridge on three singular values)
Let and . Then , and . Truncation with keeps the first two directions, so , and the approximation error comes entirely from the discarded direction. Ridge with has filter values , and , hence . The noise bounds are for truncation and for ridge, while the actual ridge amplification on this is . If the third coordinate were genuine signal, both methods would lose almost all of it: the error is for truncation and about for ridge.
Common mistake
(A zero singular value and a small one are different)
A zero singular value leaves the corresponding coefficient undetermined by the residual: every value gives the same fit, and the minimum-norm solution sets it to zero. A small positive singular value determines the coefficient as , so noise in becomes in the solution. Replacing a small positive by zero is a modeling decision that changes the problem; it is not an identity.
Common mistake
(The uniform ridge bound is not the actual amplification)
The bound holds for every and is attained only when some . For and it gives , while the actual factor is . Quoting the uniform bound as the amplification overstates the noise sensitivity whenever no singular value lies near .
Practice with solutions
Quick check
(For and , find all least-squares minimizers and the minimum-norm one.)
Compare the residual of for different .
Solution · Answer
Every minimizes the residual, whose norm is ; the minimum-norm choice is .
Quick check
(For the same and , compute .)
Use the filter on the one nonzero singular value.
Solution · Answer
because .
Quick check
(Why does the ridge noise-amplification bound grow as ?)
The bound is .
Solution · Answer
The bound tends to infinity. This is a uniform upper bound over singular values. For a fixed nonzero , the actual factor stays below and tends to as .
Quick check
(If exceeds every singular value, what is ?)
Count the terms kept in the sum.
Solution · Answer
The retained sum is empty, so .