Motivation
A finite set of basis functions turns approximation of an unknown function into estimation of finitely many coefficients. The difficult choice is the space in which those coefficients live. A polynomial of moderate degree may be effective for one variable and already have an impractical number of coefficients in many variables. A low dimensional linear space learned from representative solution snapshots can be small even when each snapshot has thousands of coordinates. These are different answers to the same question: which restricted family can represent the objects we care about to a stated accuracy?
The existing regression and principal component notes study fitting and matrix coordinates. Here we add the function-approximation perspective: the probability measure defines the error norm, polynomial index sets determine complexity, and proper orthogonal decomposition (POD) chooses a spatial subspace from weighted snapshots. We do not assume that small sample reconstruction error proves small error on every possible future state. A bridge from representative samples to an entire function family needs additional assumptions.
The main prerequisites are inner products, orthogonal projections, eigenvalues of real symmetric matrices, and elementary probability. We separate a constructive theorem on a cube from a finite-data optimality theorem. Neither theorem says that every smooth high dimensional function is cheap to approximate. Their hypotheses and their costs explain precisely the opportunities and limitations.
Definitions
Let be a probability measure on a domain . The space identifies functions equal almost everywhere and uses the inner product and squared norm . A finite-dimensional subspace has an orthonormal basis after zero-norm redundancies are removed. Its orthogonal projection is .
The measure matters. Uniform approximation controls the maximum pointwise error, whereas an norm weights errors according to . A function can have small average squared error and a large error on a set of small probability. Fitting values at finitely many nodes defines another, empirical norm. If distinct functions agree at every node, that empirical quantity is only a seminorm on their ambient function space.
For variables, a multi-index has nonnegative integer coordinates, total degree , and monomial . The full tensor polynomial space has in every coordinate and dimension . The total-degree space has and dimension . These are distinct spaces. Reducing the index set reduces cost but also removes possible approximants.
For a continuous function on , its tensor Bernstein polynomial of coordinate degree is
Here denotes the vector of divided coordinates. For each fixed , let be independent binomial random variables with parameters . The basis weights are their joint probabilities, so . This probabilistic identity is a proof device; evaluating the polynomial does not require random simulation.
For POD, take snapshots and strictly positive weights summing to one. Form and . A rank orthogonal projector represents an dimensional subspace, where . The mean reconstruction error is . This version is uncentered: its subspaces pass through the origin.
Theorem/proposition
Theorem 8.1.1
(Projection optimality and a constructive polynomial bound)
For and finite-dimensional , every satisfies
Thus is the unique best approximant as an element of . If on is Lipschitz with constant in Euclidean distance, then
The Bernstein polynomial uses up to sampled values. The error bound establishes convergence for each fixed , not a dimension independent computational cost.
The second theorem replaces a fixed function space by a linear subspace chosen from data.
Theorem 8.1.2
(Weighted snapshot POD optimality)
Let the eigenvalues of be . Among all rank orthogonal projectors, projection onto eigenvectors with the largest eigenvalues minimizes the mean snapshot reconstruction error. Its minimum is
For , a strict gap makes the minimizing subspace unique. A repeated eigenvalue across the cutoff allows multiple minimizing subspaces. These conclusions concern the specified snapshots, weights, Euclidean norm, and linear subspaces.
Concept lensAlgebraic
POD as best low-rank approximation of the snapshot matrix
The nonzero eigenvalues of are the squares of the nonzero singular values of , and the POD modes are its leading left singular vectors. The theorem is therefore a matrix statement: among all matrices of rank at most , the minimum of is , attained by . Indeed, once the column space of lies in the range of a rank projector , the closest such matrix is , column by column.
Proof sketch or proof idea
Proof
(Orthogonal residual and the binomial representation)
The residual is orthogonal to every basis function and hence to . The two terms in are perpendicular, so expanding the squared norm gives the projection identity. The second term vanishes only when in , which is the uniqueness claim.
For the polynomial bound, the probabilistic representation and the Lipschitz assumption give
The middle inequality is Cauchy–Schwarz, and the equality uses and . Each summand is at most , so the right side is at most for every .
In a nonorthogonal basis the same orthogonality argument produces a Gram linear system. Ill-conditioned coordinates can make that system numerically difficult even though the best approximating function is well defined. The Bernstein argument gives a safe rate, not the sharpest possible rate for every smoother function: polynomials reproduce affine functions exactly, and additional structure can improve actual errors greatly.
Proof
(Trace form of the weighted snapshot error)
Because is symmetric and idempotent,
Let be an orthonormal eigenbasis with , and put . Then , each lies in because an orthogonal projection does not increase length, and . The cases and are immediate, so let and set . Using for and for ,
The projector onto has and attains equality, so the minimum error is . If , equality forces , hence and for every . These vectors span the dimensional range, so the minimizing subspace is unique.
Proof X-ray
Where the projector constraints enter
The proof never optimizes over subspaces directly. The trace identity turns the error into a linear function of the numbers , and all the geometry of is compressed into two constraints: and . Both use that is an orthogonal projector of rank . An oblique projector can have , and the bound would then fail.
The key move is the exchange count . Whatever weight places below the cutoff must be missing from positions above it, and each such unit of weight changes the retained energy by at most . The argument closes in two separate ways: attainment by the leading eigenvectors gives the minimum value, and the factor gives uniqueness only when it is strictly negative. At a tie this factor is zero, is unconstrained, and different subspaces inside the tied eigenspace have the same error.
If the ambient inner product is for positive definite , first transform snapshots by . Euclidean POD in transformed coordinates then yields modes orthonormal in the inner product after transforming back. Applying the Euclidean formula to untransformed coordinates silently solves a different problem.
For large spatial dimension and a smaller snapshot count, the same nonzero eigenvalues can be obtained from the snapshot Gram matrix . If with unit vector and strictly positive eigenvalue, then is a unit eigenvector of . Indeed its squared norm is , and multiplying by gives the eigenvalue equation. This provides a practical way to construct modes without storing the larger covariance matrix. Zero eigenvalues cannot be used in this division, and modes beyond the snapshot rank are not identified by the data.
The polynomial projection has a related computational caveat. Exact inner products require integration against the chosen measure. Replacing those integrals by quadrature or sample averages produces a different finite problem. Its Gram matrix can become singular when too few distinct nodes are used, even though the continuous polynomial basis is independent. More nodes alone are not enough if they are placed where important basis functions vanish or become indistinguishable. Numerical integration accuracy, sample geometry, and the original approximation error are therefore separate parts of a reliable calculation.
Worked examples
Worked example 8.1.3
(Best affine approximation is not interpolation)
Use the uniform probability measure on and approximate by . Orthogonality against one and gives and , because and odd moments vanish. The minimum squared error is . Interpolating at the two endpoints instead gives the constant one, with error . Both constructions are mathematically valid, but they optimize different criteria. The best average approximation need not pass through either observed endpoint.
Worked example 8.1.4
(An exact Bernstein calculation and a dimension count)
For on , the binomial second-moment identity gives . The uniform error is exactly , reached at . This is faster than the general Lipschitz bound because this particular function has additional smoothness. In six variables with coordinate degree seven, the tensor space has coefficients. The total-degree-seven space has . This large saving does not make the spaces interchangeable: the tensor monomial is absent from the total-degree-seven space.
Worked example 8.1.5
(POD depends on the snapshot weights)
Take , with equal weights. Then . The best one dimensional subspace is the horizontal axis, and its mean squared error is . Changing the weights to , gives ; now the vertical axis is optimal and the minimum error is . A mode ranking is therefore a conclusion about a specified weighting of the data, not an intrinsic ranking of coordinate directions.
If the equal-weight snapshots are centered first, their centered values are and its negative. They lie on one line and have zero rank-one centered reconstruction error. Adding the mean back produces an affine line through both original snapshots, not the origin-passing line used in uncentered POD.
Common mistakes
Common mistake
(Quoting a degree instead of a dimension)
“Degree seven” must specify whether it means coordinate degree or total degree, and an error estimate in terms of degree alone can conceal an exponential coefficient count. Sparse index sets, low-rank tensor models, and additive structure can help, but each requires a separate approximation argument for the class under consideration.
Common mistake
(Reading average optimality as a per-snapshot guarantee)
POD minimizes a weighted sum, so a rarely weighted state can have a large residual. A future state orthogonal to all retained modes can be missed completely. Even zero error on a snapshot set gives no uniform bound on states absent from that set.
Common mistake
(Changing the centering or the norm without noticing)
Centering changes the approximation family from an origin-passing subspace to an affine family after the mean is restored, and a different physical norm changes the optimizer as well. Close eigenvalues can also make individual modes unstable under perturbations; an accurately reconstructed subspace may be more meaningful than the orientation of a single nearly tied eigenvector.
Counterexample mode
The best average fit is not the best worst-case fit
Claim: the projection also minimizes the maximum error. Take uniform on , and affine approximants. The projection is the constant , whose error reaches at . The constant has error , so its maximum error is ; its mean squared error is . The repair: projection is optimal only in the norm that defines it, and a uniform guarantee needs a uniform argument, such as the Bernstein bound.
Summary
Projection chooses the best function within a fixed space for a specified norm. Bernstein polynomials provide a concrete approximation construction and expose its tensor sampling cost. POD instead adapts a linear subspace to weighted snapshots, with a precise eigenvalue-tail formula for average reconstruction error. The unresolved question in an application is whether the chosen polynomial structure or snapshot family represents the unseen functions of interest.
Exercises
Quick check
(For uniform on , find the best affine approximation to and compute its squared error.)
Use the orthogonality conditions, not interpolation.
Solution · Answer
Write the approximation . Orthogonality to one gives . Orthogonality to gives . Its squared error is . Coefficient uniqueness follows because one and have positive, mutually orthogonal norms.
Quick check
(Let on . Evaluate its tensor Bernstein polynomial exactly and compare its actual error with the general Lipschitz bound.)
Explain why a bound need not be attained.
Solution · Answer
By linearity and , the Bernstein polynomial is exactly . Its error is zero. A Euclidean Lipschitz constant is , so the general bound is . A nonzero upper bound is consistent with zero actual error: the proof used inequalities valid for a much larger class and did not exploit affine reproduction.
Quick check
(Snapshots , , have equal weights. Find the rank-one and rank-two minimum mean squared reconstruction errors. Is the rank-one minimizing subspace unique?)
Compare the eigenvalues on both sides of each cutoff.
Solution · Answer
The covariance is . The minimum rank-one error is , and the minimum rank-two error is . Every line in the plane of the first two coordinate directions is a rank-one minimizer because the leading eigenvalue is repeated. The rank-two subspace is unique because at its cutoff.
Quick check
(A snapshot covariance matrix has eigenvalues . Find the smallest rank retaining at least of its total energy. Why is this not a guarantee for every state?)
Energy is a weighted aggregate.
Solution · Answer
The total energy is . Rank one retains , while rank two retains , so two is the smallest rank. The discarded energy is in the weighted aggregate. A state pointing entirely along the third eigenvector has all its energy discarded, illustrating why an average energy criterion does not imply a uniform per-state fraction.