From geometry to equations
A subspace may be easy to describe by vectors that are not orthogonal. For instance, the natural basis for affine functions is , although their inner product on is not zero. Orthogonalizing is one option; another is to translate the condition that the error is perpendicular to the subspace into a finite system of equations. This gives a concrete way to use the projection theorem.
Let be a real or complex inner product space, with the inner product linear in its first argument. Fix and . Even if is incomplete, this finite-dimensional subspace is complete and closed in the inherited norm. For , write a candidate approximation as .
Definition 4.3.1
(Gram matrix and normal equations)
For and , the Gram matrix and the vector are
The linear system in the unknown is the system of normal equations for approximating from .
The order of the indices respects our first-slot-linear convention. Since , the residual is perpendicular to every vector in exactly when for each , or equivalently . In complex spaces, changing the inner-product convention changes where conjugates occur; one should derive the equations from orthogonality rather than memorize an index order.
Common mistake
(Copying the real formulas into a complex space)
Writing instead of conjugates the right-hand side and gives a wrong vector. In with , take and . Then , while and . The wrong choice gives and , so the residual is not perpendicular to . The correct right-hand side is computed in the third worked example below.
Why the system works
Proposition 4.3.2
(Normal equations and best approximation)
Let , and . The normal equations always have a solution. Every solution represents the same vector , and is the unique best approximation to from :
Proof
(Existence through an orthonormal basis, uniqueness through Pythagoras)
Choose a basis of from the given vectors and apply finite-dimensional orthogonalization, obtaining an orthonormal basis of . Put . Then for each , so . Expressing in the original spanning vectors gives coefficients with , and the orthogonality relations say exactly that .
Conversely, every solution of gives with . Then lies in and is perpendicular to , hence and . Finally, for any the vector lies in and is perpendicular to , so Pythagoras gives
which shows that with equality only for .
Proof X-ray
Where consistency comes from
The proof never solves the linear system directly. Solvability is imported from geometry: the orthonormal basis produces a vector with , and only afterwards is rewritten in the spanning vectors. This matters when is singular, because a singular square system is not solvable for every right-hand side. The special right-hand side is what makes it consistent. Uniqueness is a separate step that uses only the definiteness of the inner product: a vector of perpendicular to is perpendicular to itself. Both steps rely on being finite-dimensional, which is why completeness of is never needed.
The second question is when the coefficients, and not just the vector , are determined. Write for .
Proposition 4.3.3
(Nullspace of the Gram matrix)
For every ,
Consequently if and only if , and is invertible precisely when are linearly independent.
Proof
(The quadratic form of the Gram matrix)
Using linearity in the first slot and conjugate linearity in the second,
If , then , so . Conversely, if , then for each . Thus the nullspace of consists exactly of the coefficient vectors of linear relations among . A square matrix is invertible exactly when its nullspace is zero.
A singular matrix therefore indicates redundant coordinates, not failure of best approximation. Coefficients may be nonunique while the approximating vector remains unique.
Counterexample mode
A singular Gram system with an arbitrary right side
One might expect that the normal equations are solvable because is a Gram matrix, whatever the right-hand side. Take with . Then , and the system asks for and simultaneously, which is impossible. The repair is that must arise as . For every in the nullspace of we then have , so is orthogonal to the nullspace of the Hermitian matrix and therefore lies in its range. Here is not orthogonal to , so no vector produces it.
The identity also gives the system a second reading, as a finite least-squares problem.
Concept lensAlgebraic
The Gram matrix as V-star V
Regard , , as a linear map, with carrying . Its adjoint is , since . Then , so
These are the normal equations of the least-squares problem of minimizing over . The nullspace statement becomes , and the vector is the projection of onto the range of .
Worked examples
Worked example 4.3.4
(Fitting an affine function to )
In real , take , , . The entries are integrals, so
The determinant is . Subtracting half the first equation from the second gives , so and . Thus the closest affine function in the norm is . Since ,
Here justifies the middle expression. The functions represent equivalence classes; continuous representatives are used to display the calculation.
Common mistake
(The L2-closest line is not the uniformly closest line)
The function above minimizes the mean-square error, not the maximum error. At its error is . The line has error , whose values at , and are , , ; since the error is convex with its minimum at , its maximum absolute value on is , smaller than . Best approximation depends on the norm.
Worked example 4.3.5
(A dependent spanning list: )
Keep in , but use , , . From , , and ,
The relation gives , which one checks row by row: , , . The vector solves the system because the first two rows are those of the previous example and the third row is their sum. Hence the full solution set is
and every such gives the same function . The matrix is singular, the system is consistent, and is unchanged.
Worked example 4.3.6
(A complex example where the conjugate matters)
In with , approximate by multiples of . Here and . Hence and
Check: . The squared error is .
The explorer below solves the normal equations exactly for and a chosen spanning set, and shows that a dependent list such as makes singular and non-unique while stays the same.
Practice with solutions
Quick check
(Find the best approximation to from in , and its squared error.)
The matrix is the same as in the first worked example; only changes.
Solution · Answer
The right side is . Subtracting half the first equation from the second gives , so and . Thus . Its squared error is .
Quick check
(Let . Describe the solutions of the normal equations and their common approximation to .)
Write out both rows of .
Solution · Answer
Both equations reduce to , so the solutions form the line . Every solution gives the same vector .
Quick check
(Why does replacing by leave the best approximation unchanged, although the matrix entries change?)
Compare the two spans.
Solution · Answer
The lists and have the same span . The best approximation from is unique, so it is the same vector ; only its coordinates change, from to .