The singular value decomposition does more than factor a matrix. It separates directions that a map stretches, directions it forgets, and output directions it cannot reach. Those distinctions explain both least squares and the unique minimum-norm solution. They also show why dividing by every singular value is wrong when the map has a kernel.
We use Euclidean or standard Hermitian norms, not an unspecified weighted metric.
Where the singular vectors come from
Let and let denote conjugate transpose. The matrix is Hermitian and positive semidefinite because
Definition 4.2.1
(Singular values and singular vectors)
Let be an orthonormal basis of eigenvectors of , with eigenvalues ordered so that the positive ones come first. The numbers are the singular values of ; write for the positive ones, with all zero values afterwards. The are right singular vectors. For the vectors are left singular vectors.
The spectral theorem supplies the basis .
Theorem 4.2.2
(Singular value decomposition)
In the notation above, is orthonormal, for and for . If is extended to an orthonormal basis of , and contain the respective basis columns, then
where is by with on its rectangular diagonal and zeros elsewhere.
Proof
(Orthonormality of the and the matrix identity)
For ,
because . If , then , so ; the norm identity gives this and does not give a left singular vector by division. Thus on the basis , and since is unitary, .
Over the reals, the same construction uses transpose and orthogonal matrices.
The positive right singular vectors span ; the positive left singular vectors span . Their complements encode different spaces: the remaining right directions are in the input kernel, while the remaining left directions form . Unequal input and output dimensions make this distinction particularly important.
The closest achievable output
For a given , minimizing means finding the closest point to in the image.
Theorem 4.2.3
(Least-squares minimizers)
Write with and . Then for every ,
A minimizer of exists. A vector is a minimizer if and only if , if and only if , that is, if and only if .
Proof
(Pythagoras in the output space)
The vector lies in and is perpendicular to it, which gives the displayed identity. The right side is smallest exactly when , and such an exists because belongs to the image. Thus the optimal output is uniquely , while the input producing that output may not be unique.
For the normal equations, a vector is perpendicular to every exactly when its adjoint image is zero: vanishes for all iff . Apply this to . Then is a minimizer iff iff iff . This proves necessity and sufficiency without differentiating a complex-valued coordinate expression or assuming an invertible normal matrix.
Proof X-ray
Why the decomposition of b does the work
The proof has one idea: split once, into a part in the image and a part perpendicular to it, and then every candidate only affects the first part. The perpendicularity is used exactly once, to kill the cross term in ; without it the identity would carry a term that depends on . Membership is used exactly once, to make vanish for some , which is why a minimizer exists even though itself may be unreachable. The finite dimension enters only through the existence of the decomposition. The statement about is not a second argument: it re-expresses the perpendicularity of to the image, so it is equivalent, and nothing about invertibility of was needed.
If the columns are independent, is positive definite and invertible. If they are dependent, writing is invalid, although least-squares minimizers still exist. A singular normal matrix is a failure of that inverse formula, not a failure of the minimization problem.
Common mistake
(Using the normal-equation inverse when columns are dependent)
For the columns are dependent and has determinant , so does not exist. Yet minimizers of exist for every (here is a number, and is solvable). Only the formula fails; the correct tool is the pseudoinverse defined below.
Recover the minimum-norm input
Definition 4.2.4
(Pseudoinverse)
With the singular vectors above, the pseudoinverse of is
Only positive singular values are inverted. For the zero matrix the sum is empty and .
Quick check
(Find for .)
Use : identify , , .
Solution · Answer
Here , so , , and . Hence . The zero singular value is not inverted; it contributes nothing.
Theorem 4.2.5
(Minimum-norm least-squares solution)
The pseudoinverse satisfies
Every least-squares minimizer has the form with , and is the unique minimizer of smallest norm. In particular does not depend on the choice of singular vectors, including choices within repeated singular-value spaces.
Proof
(Projections, then the Pythagorean identity in the input space)
From we get . From we get , so . The first sum projects onto the span of , which is ; the second projects onto the span of , which is . The first projection selects the closest achievable output, the second removes the input component that the map cannot detect.
By the previous theorem, is a minimizer iff iff iff with . The vector belongs to , so
which is smallest exactly at . The minimum-norm minimizer is therefore . Since this description mentions only and , and not the singular vectors, the linear map is the same for every admissible choice.
Quick check
(For and , find the minimum-norm solution of .)
Compute for a single nonzero row.
Solution · Answer
Here , , , so and . Check: , and every other solution has norm squared .
Concept lensStructural
The pseudoinverse as the inverse of a restricted map
There is a basis-free description. The restriction of from to is a bijection: it is injective by the zero intersection with the kernel, and surjective by decomposing any preimage into its kernel and perpendicular parts. First project onto the image, then apply the inverse of this restriction. The result is precisely . This explains why the pseudoinverse exists for every finite-dimensional map even when an ordinary inverse does not exist, and why its definition involves two different spaces rather than one square matrix.
The projection identities imply and , with both products and self-adjoint. These are the four Moore–Penrose identities. They describe orthogonal projections rather than an ordinary two-sided inverse. For the zero matrix the sums are empty, the pseudoinverse is zero, and the same conclusions remain valid.
Counterexample mode
The reverse-order law for products
For invertible matrices , and one might expect . Take and , so and . A single nonzero row or column has , so and . Then . Neither factor is invertible, so the inverse law gives no guide here: the kernel of and the image of are unrelated subspaces, and the projections they define do not line up. The pseudoinverse of a product must be computed from the product itself, not assembled from the factors.
Worked example 4.2.6
(A full rectangular factorization with rank loss)
Let
Construct a full real SVD, find all least-squares minimizers and prove which one has minimum norm.
Solution. The norms are , . The only positive singular value is . Take
and
Each list is orthonormal by direct dot products. With those columns in and
we have . No vector was divided by the zero singular value: the remaining left basis vectors were chosen as an orthogonal completion.
The pseudoinverse is
The image projection is , so the closest output is . The residual is
Its dot product with is zero, verifying the normal equations and optimality.
The kernel is . Therefore all minimizers are
The two summands are perpendicular, giving
The unique minimum-norm solution occurs at . This separates uniqueness of the fitted output from nonuniqueness of the fitting input.
Worked example 4.2.7
(Exact solvability does not give a unique input)
Let
Find every exact solution and the unique one of minimum norm.
Solution. The equations give for real . Their squared norms satisfy
Thus the minimum-norm input is . The kernel is spanned by , and the chosen input is perpendicular to that vector, independently verifying the characterization.
Here the rows are independent. Hence
Multiplying gives . The inverse exists for , not for the singular three-by-three matrix . Also , while is a rank-two projection rather than .
The output can be fitted exactly because the image is the whole output space. The input remains nonunique because the kernel is nonzero. A zero residual therefore settles solvability but does not settle uniqueness. The minimum-norm rule supplies a distinguished solution through orthogonality, without pretending the original map is a bijection.
The coordinate proof shows exactly what is free
Write and . Unitary invariance of the norm gives
The coordinates do not appear. Set for positive singular directions and leave the kernel coordinates arbitrary. Setting those free coordinates to zero gives the smallest input norm.
This proof is valid for tall, square and wide matrices, including rank-deficient ones. It does not count output directions as extra input variables. Nor does it require to lie in the image. The unused output coordinates determine the unavoidable residual, while unused input coordinates determine the family of equally good fits.
Common mistake
(A small residual does not certify an accurate input)
Some input directions may be nearly invisible to the map; a large change there can have a small effect on the output, so a small residual alone cannot show that an input is accurate. The third worked example below makes this quantitative: the data change by while the solution changes by and the residual does not change at all.
Theorem 4.2.8
(Norm of the pseudoinverse)
For a nonzero map with smallest positive singular value ,
Proof
(Estimate in left singular coordinates and attain equality)
In the left singular coordinates ,
Equality is attained at the left singular vector belonging to , for which . Thus the estimate is sharp, not merely a convenient upper bound. Applying the same inequality to a data difference gives for this fixed map.
The projection itself does not amplify lengths: its norm is at most one. Amplification occurs while recovering the input coordinates, where division by a small positive singular value is unavoidable. In contrast, the discarded kernel coordinates are assigned zero by the minimum-norm convention. These are different operations. The zero map has pseudoinverse zero and norm zero, so no smallest-positive-value formula is used in that separate case.
Worked example 4.2.9
(A small data change with an unchanged residual)
For , let
Compare the minimum-norm least-squares inputs and explain what happens at .
Solution. The singular values are one and , so
The two inputs are and , respectively. Both residuals are with norm one. Yet the data change has norm , while the input change has norm one. Letting tend to zero exposes amplification by .
At zero the rank changes. The second input direction is now a genuine kernel direction; it must be assigned zero, rather than divided by zero, in the pseudoinverse:
For the limiting data , all fitting inputs have first coordinate one and arbitrary second coordinate. The minimum-norm choice is . The perturbed positive-parameter inputs do not converge to that minimum-norm choice, even though their data converge to .
This is not a contradiction. Pseudoinversion is sensitive near rank changes. On a fixed positive singular direction, errors are scaled by its reciprocal; when the singular value becomes exactly zero, the geometric rule removes that direction instead.
Quick check
(Suppose . Is the least-squares output unique, and is the input unique?)
Separate the roles of the image and the kernel.
Solution · Answer
The output is always unique: it is the projection of onto . The input is unique if and only if , because the minimizers form the set .
What is invariant and what remains a choice?
Singular values are determined by , but singular vectors need not be unique. Repeated positive singular values permit simultaneous orthonormal changes in their associated input and output subspaces. Kernel and left-nullspace bases can also vary. The products defining the two orthogonal projections and the minimum-norm solution remain unchanged.
Finally keep the norm hypothesis visible. Weighted least squares uses the chosen input/output metrics and their associated adjoints. The standard formulas above cannot simply be transplanted while retaining an ordinary transpose. This chapter treats standard Euclidean/Hermitian geometry; numerical algorithms, truncation thresholds and weighted implementations are further topics, not hidden assumptions in these exact proofs.