Coordinates as approximation
In Euclidean space an orthonormal basis represents a vector by finitely many independent coordinates. In a Hilbert space there may be infinitely many coordinates. Their finite partial sums still have a precise geometric meaning: each is a best approximation from a finite-dimensional subspace. The extra question is whether the approximation error tends to zero as the subspaces grow. An orthonormal system alone does not ensure this; a vector can be perpendicular to every member of the system and still be nonzero.
Throughout, is a real or complex Hilbert space, and is linear in its first argument. We restrict attention to a finite or countably infinite system; the results below do not assume that every Hilbert space has a countable orthonormal basis.
Definition 4.1.1
(Orthonormal sequence, coefficients and partial sums)
A sequence in is orthonormal if for and for . For its coefficients are , and its th partial sum is
Definition 4.1.2
(Complete orthonormal system)
The closed linear span consists of the limits in the Hilbert norm of finite linear combinations of the . The system is complete in if .
Common mistake
(A complete system is not the same as a complete space)
Completeness of is a property of the normed space: Cauchy sequences converge. Completeness of the system is a property of the chosen vectors: their closed span is all of . Every orthonormal sequence in a Hilbert space lives in a complete space, yet the sequence in is not a complete system there. The two words play different roles in the proofs below.
Finite-dimensional best approximation
Proposition 4.1.3
(Partial sums are best approximations)
For every and , is the unique closest point to in . Moreover,
Proof
(Orthogonal residual and Pythagoras)
For , orthonormality gives , so the residual is perpendicular to . If , then is an orthogonal decomposition, since . Pythagoras gives
The right side is smallest exactly when , which proves the unique minimum. Taking and expanding by orthonormality gives the stated identity.
No infinite-series theorem is used here. In particular, for any proposed coefficients , the squared error is the squared projection error plus . The inner product convention matters: with the alternative convention linear in the second argument, the coefficient would be written instead.
Worked example 4.1.4
(Least-squares approximation of in )
On , with the usual real inner product, let and . Both have norm one, since , and their inner product is . For ,
Thus the best affine approximation in the norm is . Its squared error is
This minimizes integrated squared error, not the maximum pointwise error. Elements of are equivalence classes modulo equality almost everywhere; the displayed continuous representatives make the computation concrete.
Bessel's inequality and convergence of partial sums
Theorem 4.1.5
(Bessel's inequality)
For any and any orthonormal sequence ,
Proof
(Bounded partial sums of a nonnegative series)
Every partial sum of the nonnegative terms is at most by the finite identity, because . A bounded increasing sequence of real partial sums converges, and its limit is still at most .
Bessel's inequality also controls the vector sums, and this is where completeness of the space enters.
Theorem 4.1.6
(The orthogonal expansion converges to the projection)
For every , the series converges in the norm of , and its sum is the orthogonal projection of onto :
Proof
(Cauchy partial sums and an orthogonal residual)
If , orthonormality yields
The right side tends to zero uniformly in as , since the scalar series converges by Bessel's inequality. Hence is Cauchy. Completeness of gives a limit . Each lies in , and is closed, so . For any fixed , the residual is perpendicular to for ; continuity of the inner product gives . By linearity is perpendicular to every finite linear combination of the , and by continuity to their limits, so . The projection theorem characterizes as the unique vector of whose residual is perpendicular to , so .
Proof X-ray
Three separate ingredients
The argument uses three facts at three different moments. Bessel's inequality turns a scalar statement into a vector one: the tail identity shows that the partial sums are Cauchy without knowing any limit in advance. Completeness of is used once, to produce the limit . In an incomplete inner product space the Cauchy sequence could have nowhere to converge. Closedness of and continuity of the inner product then pass two properties to the limit: membership in and perpendicularity of the residual to each . The identification needs no further computation, because a vector of with residual perpendicular to is unique. Nothing in the proof says ; that is exactly the extra information carried by completeness of the system.
The equality asserts norm convergence; it does not assert pointwise convergence when the vectors are functions. If is a proper subspace, the series converges to the projection, not necessarily to .
Theorem 4.1.7
(Riesz–Fischer construction)
For every scalar sequence with , the series converges in to a vector with for every and . This vector is the only element of with these coefficients.
Proof
(The same tail estimate with prescribed coefficients)
The partial sums satisfy for , so they are Cauchy and converge in to some , which lies in the closed subspace . For we have , and continuity of the inner product gives ; continuity of the norm gives . If has the same coefficients, then is perpendicular to every , hence to , so .
Uniqueness holds only within : a vector outside can have the same coefficients, since adding any element of does not change them.
Parseval and completeness of the system
Theorem 4.1.8
(Characterizations of a complete orthonormal sequence)
For an orthonormal sequence in , the following are equivalent:
- Its closed linear span is .
- The only vector perpendicular to every is zero.
- For every , in norm.
- For every , Parseval's identity holds:
Proof
(A cycle of implications)
The preceding theorem gives . Thus (1) implies (3), since then . Next, (3) implies (4) by letting in the finite identity . If (4) holds and is perpendicular to every , then , proving (2). Finally, if (2) holds, the vector is perpendicular to and hence to every . It is therefore zero for every , so and (1) holds.
For an individual , equality in Bessel's inequality holds exactly when : letting in the finite identity, with , shows that the difference between the two sides is . An incomplete system may thus satisfy Parseval for some vectors, but not for all of .
Counterexample mode
An orthonormal sequence missing one direction
The claim to test is that an infinite orthonormal sequence in automatically satisfies Parseval's identity. In , take the standard unit vectors and omit . Their closed span is the subspace of sequences with first coordinate zero. For , every coefficient with is zero, so every partial sum is zero and the series converges to . The left side of Parseval is , the right side is , and Bessel is strict. The repair is condition (2) of the theorem: Parseval holds for every vector exactly when no nonzero vector is perpendicular to the whole system. The omitted system is complete in its own closed span, which is why the ambient space must be stated.
When the system is complete, the coefficients become a faithful coordinate description of .
Concept lensStructural
A complete orthonormal sequence as a unitary coordinate map
Let be a complete orthonormal sequence in , and define by . Bessel's inequality says , and is linear. Parseval's identity says , so is an isometry, and by polarization . The Riesz–Fischer construction says every square-summable sequence is for some , so is onto. Thus is a unitary isomorphism of onto that sends to the th standard unit vector. Computations with coefficients are computations in .
Worked example 4.1.9
(The standard system in is complete)
For the full standard system in and , we have , so truncates the sequence after the th coordinate. Then
which is the tail of a convergent series and tends to zero by the definition of . Condition (3) of the theorem holds, so the system is complete and . This verifies completeness directly rather than inferring it merely from orthonormality.
Worked example 4.1.10
(Bessel equality for one function: the sine system and the constant )
In real , the functions , , are orthonormal: , and for the integral of is zero. For ,
which is for odd and for even . Using ,
Equality in Bessel's inequality shows , so lies in the closed span of the sine system and its sine series converges to in . This is a statement about one vector; the computation does not by itself prove that the sine system is complete.
Common mistake
(Norm convergence is not pointwise convergence)
In the last example, converges to in . Every partial sum vanishes at , because , so the series converges to at that point, not to . There is no contradiction: an identity is an equality of equivalence classes and says nothing about the value at a single point.
Practice with solutions
Quick check
(In with the usual inner product, let and . Find the best approximation to from their span and its distance to that span.)
Compute the two coefficients first.
Solution · Answer
The coefficients are , so . The span is the plane with third coordinate zero. The residual is , so the distance is ; as a check, .
Quick check
(Let be orthonormal in and let , with norm convergence. Find and .)
Identify the coefficients of first.
Solution · Answer
By the Riesz–Fischer construction, and . The residual is , so .
Quick check
(In , use only . Characterize the vectors for which equality holds in Bessel's inequality.)
Compute minus the sum of the squared coefficients.
Solution · Answer
The coefficients of are , so the difference between the two sides is . Equality holds exactly for the sequences whose first coordinate is zero, that is, for the vectors of the closed span.
Quick check
(Why does for every imply when the system is complete? Give a counterexample when it is not.)
Use density of the span and continuity of the inner product.
Solution · Answer
If is perpendicular to every , then by linearity it is perpendicular to their span, and by continuity to its closure, which is when the system is complete. In particular , so . For the system in , the vector is perpendicular to every listed vector.