When a fitted answer is sensitive
A data pipeline often seeks coefficients that make close to observed responses . Block QR can reduce the data without explicitly forming , but a small or vanishing diagonal in a triangular factor raises a further question: which combinations of coefficients are actually determined by the observations? The singular value decomposition (SVD) answers this geometrically. It also shows why making the residual small does not by itself guarantee reliable coefficients.
Let and . All norms below are Euclidean for vectors and induced Euclidean operator norms for matrices. Work first in exact arithmetic.
Definition 4.3.1
(SVD and singular coordinates)
A full SVD writes with orthogonal and and a diagonal whose positive entries are the singular values ; here is the rank of . The columns of and of are orthonormal singular vectors with and for . The singular coordinates of are . If , sums over are empty.
The coordinates of perpendicular to the column space, along , cannot be fitted by any .
Theorem 4.3.2
(All least-squares minimizers)
The minimizers of are exactly
The unique minimizer of smallest Euclidean norm is obtained with . Its squared residual is .
Proof
(Diagonalize the residual)
Expand in the orthonormal right singular basis. For , the coefficient of along is ; all other components of vanish. Hence
Every term in the first sum can simultaneously be made zero, forcing . The remaining right singular directions form , establishing the complete family. They are orthogonal to the displayed sum, so adding a nonzero strictly increases the squared norm by . The residual formula follows by orthogonal projection. This also covers : every vector fits equally well and is the unique minimum-norm choice.
The proof works because one orthogonal change of basis on each side turns a coupled problem into separate scalar ones.
Proof X-ray
Separate fitted and invisible directions
Orthogonality of splits the squared residual into independent coordinates, one for each left singular vector; orthogonality of gives independent coordinates. The key move is that in these coordinates acts diagonally, so each of the first residual coordinates depends on a single coefficient and all of them can be zeroed at once, while the remaining coordinates do not depend on at all: that part of lies outside the column space. Positivity of is what allows the division that fixes those coefficients. The kernel directions are exactly the coordinates that ignores; they do not affect the fit but add to the coefficient norm, which is why setting them to zero, and nothing else, gives the unique minimum-norm minimizer.
The same two conditions also describe the minimum-norm minimizer without coordinates.
Concept lensGeometric
Projection of $b$, preimage in the row space
Every minimizer has , the orthogonal projection of onto the column space of . The minimum-norm minimizer is the unique solution of that projected equation lying in , the row space. In operator form it is with the pseudoinverse . The geometric description, project in the output space and then take the preimage orthogonal to the kernel, needs no choice of basis inside a repeated singular value.
Worked example 4.3.3
(A rank-deficient matrix through its SVD)
Let and . Here , , and , while . Then , so the minimizers are . The minimum-norm minimizer is , and the squared residual is . Directly, the objective is , minimized exactly when .
Common mistake
(Inverting the normal matrix of a rank-deficient problem)
For the matrix of the previous example, has determinant , so the formula does not exist, although least-squares minimizers do. Writing it anyway, or replacing the inverse by whatever a solver returns for a singular matrix, selects one minimizer arbitrarily or fails. The minimum-norm choice requires an explicit rule, such as the pseudoinverse above.
Worked example 4.3.4
(A tall rank-one matrix)
Let and . Then , , and . The singular coordinate is , so the minimum-norm minimizer is . The predicted vector is , and the residual is perpendicular to , since . Its squared norm is , which agrees with .
Small singular values and truncation
A small positive singular value is harmless for the fit but dangerous for the coefficients.
Proposition 4.3.5
(Response sensitivity of the minimum-norm solution)
For a fixed with and a perturbed response , the minimum-norm solution changes by
with equality for .
Proof
(Linearity and orthonormality)
The minimum-norm solution is linear in , which gives the formula for . Since the are orthonormal, , using and Bessel's inequality for the orthonormal . For only the last term is nonzero and equals .
Thus even a small observation error aligned with can cause a large absolute coefficient error. This is a fixed-matrix bound; it says nothing by itself about errors in , or a relative error when the true solution is zero.
Definition 4.3.6
(Truncated SVD solution)
For an integer with , the truncated SVD (TSVD) solution keeps only the first singular directions:
Using rather than a numerical threshold makes the convention explicit. For , the change in caused by a response perturbation has norm at most , by the proof above with replaced by ; for this change is identically zero. The price is the omitted fitted energy in squared residual compared with the exact minimum. A potentially larger residual is therefore a deliberate trade for reduced sensitivity, not proof that the computation failed. If a cutoff divides equal singular values, the retained subspace must also be specified; another valid basis of that repeated singular subspace may give another .
Worked example 4.3.7
(A small residual can hide a fragile coefficient)
Let and . The minimum-norm solution is with zero residual. If the second observation rises by , the second coefficient rises by : an amplification factor of ten. Keeping only the largest singular direction gives , residual norm , and response-error amplification at most . It would be wrong to conclude that the discarded coefficient is truly zero: the calculation has simply chosen not to estimate it from this weak direction.
Common mistake
(Reporting a truncated coefficient as an estimate of zero)
In the example above, TSVD returns in the second coordinate. That zero records a decision not to estimate along ; it is not evidence that the second coefficient is zero, and the data are in fact consistent with the value . Report the truncation level with the coefficients, and describe the discarded directions as unestimated.
Counterexample mode
A perfect training fit is not a stability certificate
Claim: a zero residual shows that the fitted coefficients are reliable. For and , the exact solution has zero residual. Changing only the second observation by to changes the second coefficient to , an order-one change from a perturbation of size , exactly the factor of the sensitivity proposition. The repaired statement bounds coefficient changes through the smallest retained singular value, not through the residual.
A computational and statistical reading
If has full column rank, , while . Forming normal equations can therefore magnify conditioning difficulties; the condition number is a property of the problem, not a universal prediction of the error of every implementation. For rank-deficient , is singular, so the ordinary inverse formula is unavailable. A compact SVD or a rank-revealing method can expose nearly dependent directions; an implementation must still choose a tolerance relative to scale, arithmetic precision and the application's noise. Exact rank and numerical rank are different concepts.
An actual large dataset may make storing or computing its full SVD impractical. A block QR reduction, a sparse iterative solver or a carefully validated approximate low-rank method may be appropriate, depending on access patterns and resources. Their costs and approximation errors require separate analysis; this exact SVD argument is an interpretation, not a claim that a full SVD is cheap or that a particular cutoff is optimal. Regularization changes the estimator, and its usefulness depends on signal and noise, not only on the matrix.
Practice with solutions
Quick check
(For and , find the full solution, the rank-one TSVD solution and its residual norm.)
For a positive diagonal matrix the singular vectors are the standard basis vectors.
Solution · Answer
The full solution is . Keeping the first direction gives and residual , whose norm is ; this equals .
Quick check
(An has positive singular values , and . Bound the coefficient change from a response perturbation of norm at most for the full minimum-norm solution and for TSVD with .)
Divide by the smallest retained singular value.
Solution · Answer
The respective worst-case bounds are and . They are bounds on absolute change for fixed , not guarantees on unknown estimation error.
Quick check
(In the theorem, what happens if is perpendicular to the column space of a nonzero ?)
Compute the singular coordinates .
Solution · Answer
All are zero. Every minimizer lies in , the minimum-norm minimizer is , and the residual norm is . No coefficient can fit the perpendicular component.