Motivation
An orthogonal polynomial basis can make a continuous approximation problem beautifully simple: each coefficient is an inner product. A computer usually receives samples rather than exact integrals. Replacing the integrals with weighted observations produces a different geometry, and that geometry need not preserve orthogonality. This is why a well-defined approximation space can coexist with an unreliable fitted coefficient vector.
Suppose the target is a response on the interval from minus one to one. We wish to measure error in mean square with respect to the uniform probability distribution on that interval, but the observed inputs are concentrated near the origin. A slope barely changes values at those inputs and therefore becomes difficult to estimate. Increasing the number of nearly identical inputs can reduce independent noise while leaving the same fundamental lack of information about variation elsewhere.
The aim is to distinguish four effects that a single residual plot can hide: the continuous approximation residual, its aliasing into sampled coefficients, amplification caused by a weak design, and random measurement noise. We derive an exact identity that separates them. Weighted least squares appears here as a tool for approximating functions in a specified measure, rather than as a repetition of the existing regression calculation.
Definitions
Definition 11.1.1
(Orthonormal polynomial basis for the uniform measure)
Let be uniform probability measure on , so . Choose a finite polynomial space with orthonormal basis . Orthonormality means that these continuous inner products form the identity matrix. For degrees zero through two one may take
The uniform moments are zero for odd powers and for an even power . These identities verify the displayed basis and can be used in Gram–Schmidt to construct higher degrees. The choice of measure is part of the definition: these polynomials need not be orthogonal under another input distribution.
Write the continuous projection as , where , and put . Then is orthogonal to . Assume is square integrable and has specified point values at the observation nodes. Point values require this extra specification because an equivalence class alone does not determine values at isolated points.
Definition 11.1.2
(Weighted sampled fit and discrete Gram matrix)
Take fixed nodes , positive weights summing to one, and independent measurement errors with zero mean and common finite variance . The observations are . Define the sample matrix , weight matrix , and discrete Gram matrix . When is positive definite, the weighted fit is and .
For a coefficient vector , continuous orthonormality gives , while its squared sample norm is . Thus eigenvalues of compare two distinct ways of measuring the same polynomial. Polynomial degree alone does not describe this comparison. In variables a total-degree space of degree at most has basis functions, and at least that many effective observations are needed for full column rank.
Theorem/proposition
Theorem 11.1.3
(Discrete design stability)
If has smallest eigenvalue at least , then every change of sampled responses satisfies
If all eigenvalues lie in with , continuous and discrete squared norms differ by at most the corresponding factors and . Full column rank is necessary for unique coefficients. Repeated observations do not restore a direction that vanishes at every sampled node.
Theorem 11.1.4
(Approximation, sampling alias, and measurement variance)
Under the fixed-design assumptions and positive definite , let and . Then
For equal weights , the last term simplifies to . If additionally the eigenvalues lie in , that variance term lies between and . No Gaussian noise assumption is required.
Proof sketch or proof idea
Proof
(A Rayleigh-quotient argument)
Set and . The coefficient perturbation obeys . Taking its inner product with gives . If the left norm is nonzero, divide by it; if it is zero, positive definiteness forces . Since , the stability bound follows. The norm-equivalence claim is the Rayleigh quotient bound for .
Proof
(Projection, alias and noise)
Insert into the fitting equation. This yields . The noise term has mean zero and covariance . Its mean squared Euclidean norm is the trace of this covariance. Continuous orthogonality gives the Pythagorean identity between the residual and the polynomial coefficient error. Averaging and using the zero noise mean proves the second theorem.
Proof X-ray
Which orthogonality produces which term
The identity rests on two different orthogonalities, used in different places. Continuous orthogonality of to under gives the Pythagorean split ; it is also what turns a coefficient error into a function error with no constant. The sampled fit, however, projects with the discrete weights, and is generally not orthogonal to in that inner product; this failure is exactly the alias vector , which is deterministic.
The cross term between and the noise term disappears only after averaging, because the design is fixed and the errors have mean zero; no independence between and the noise is needed, since is not random. Independence and equal variance enter only in the covariance , and no Gaussian assumption is used anywhere.
For equal weights, , yielding the simpler covariance . Summing inverse eigenvalues gives the stated bounds. More generally, the inequality gives an upper variance bound . This shows why strongly concentrated weights can reduce the effective averaging even if their sum remains one.
Concept lensGeometric
One residual, two different orthogonality tests
The residual is perpendicular to every polynomial in under integration against . It need not be perpendicular under the finite weighted sum. The vector measures that discrepancy. Therefore a noiseless sampled fit can have biased continuous coefficients even when the discrete linear system is solved exactly. Calling a basis orthogonal is incomplete until the inner product has been named.
Computationally, form the weighted matrix and response , then solve by a rank-revealing QR factorization or singular value decomposition. The normal equations describe the minimizer but square the matrix condition number if used as the numerical solution method. Scaling to an orthonormal continuous basis helps interpret the problem; it does not replace checking the sampled matrix's singular values.
The same geometry explains why pointwise prediction uncertainty can be uneven. Put . Cauchy–Schwarz gives . Under equal-weight independent noise, pointwise fitted variance is , at most . The continuous average of is , but its value at a particular point can be much larger. For the three displayed polynomials, its endpoint value is nine. A stable integrated fit therefore need not have equally precise predictions at every location.
Complexity becomes concrete in several variables. With twenty inputs, a total-degree-two polynomial already has 231 coefficients, while degree three has 1771. Two hundred observations cannot uniquely identify all degree-two coefficients without further restrictions, regardless of the solver. Repeating those observations at the same nodes does not enlarge the row span of the design. It can average down independent measurement noise, but a polynomial that vanishes at every distinct node remains invisible. Additional structure or a penalty can select a fitted function, yet it does not turn the original observations into information about that missing direction.
Worked examples
Worked example 11.1.5
(A continuous degree-two approximation)
Approximate in the span of the three displayed basis functions. Its coefficients are , , and . Hence . The squared approximation error is . This is an exact integrated error, not a training residual. Computing it required the specified uniform measure, and fitting at unrelated nodes need not recover these same coefficients.
Worked example 11.1.6
(A three-node rule that preserves degree-two geometry)
Choose nodes with weights . The weighted odd moments vanish, while the weighted second and fourth moments equal and . Therefore products of the displayed basis functions have exactly their continuous inner products, and for the degree-two space.
For a target inside that space, the sampling-alias term is zero. With independent errors of common variance , the integrated fitted variance is , despite the unequal weights. To check this directly, the square matrix is orthogonal, so . The covariance trace is then . This calculation uses the general weighted formula, not an unjustified equal-weight substitution.
Worked example 11.1.7
(Two nodes can identify a slope very poorly)
Use only and take equal-weight nodes , where . Then . For the target , the noiseless fitted polynomial is the constant , whereas the best continuous affine approximation is the constant . Thus the alias contribution is and the continuous residual contribution is .
With two independent noisy measurements, the exact expected error is . As the nodes approach zero, the slope variance diverges. Repeating a large number of such near-zero measurements reduces noise according to their count, but a fixed small spatial spread still makes slope estimation expensive. At , the slope is not identified at all; the invertible formula no longer applies.
Common mistakes
The second theorem separates the alias from stability, and the two really are different.
Counterexample mode
A unit discrete Gram matrix does not remove sampling alias
Claim: if the weighted nodes reproduce the continuous geometry of , so that , then the noiseless sampled fit equals the continuous projection. Take the three-node rule of the second worked example, for which , and the target of the first. The sampled values are . With the fit is : odd symmetry gives , the rule gives , and
The fitted polynomial is , not . The alias is , about the same size as , and the total error agrees with a direct integration of . The rule integrates polynomials exactly only through degree five, while has degree six. The correct statement is the theorem: controls stability and variance, whereas the alias vanishes only when the weighted sum also reproduces the inner products of with the basis, for instance when .
Common mistake
(Adding polynomial terms only lowers the error)
More polynomial terms do not merely decrease approximation error. They enlarge the sampled matrix, can introduce near dependencies, and increase the variance trace. The simple noise scale requires a well-conditioned, equal-weight design; it is not a universal formula for every polynomial fit. Choosing degree from the same observations also changes the estimator, so the fixed-space calculation must then be supplemented by a selection analysis.
Common mistake
(Exactness on basis products is not exactness on the residual)
Exactness of a quadrature rule on basis products controls , but it does not automatically control products involving an arbitrary target residual. Consequently it need not eliminate sampling alias, as the counterexample above shows. A function can vanish at every quadrature node and still be nonzero almost everywhere between them. Neither exact arithmetic nor a zero training residual resolves that missing information.
Common mistake
(Assuming equal noise variances without saying so)
Finally, equal observation variances were a modeling assumption. If variances differ, replace by their covariance matrix in the derivation. Inverse-variance weighting may be sensible for estimating certain coefficients, but the intended continuous error measure and the sample geometry still need to be specified. A weight chosen for one purpose does not automatically optimize every other purpose.
The formulas also assume that the recorded inputs are the true design locations. Errors in input coordinates change the basis evaluations themselves, rather than only the response vector. That situation needs an additional model and is not covered by the response-noise identity.
Summary
Continuous orthogonality makes coefficient error equal function error inside the approximation space. Sample fitting introduces a discrete Gram matrix whose spectrum controls stability. The exact expected error separates continuous approximation, deterministic sampling alias, and measurement variance. Reliable polynomial approximation therefore needs a suitable space, informative nodes, appropriate weights and a stable linear solver, rather than degree selection alone.
Exercises
Quick check
(Verify that is orthogonal to both one and , and that its squared norm is one under uniform probability measure on . Identify every moment used.)
Solution · Answer
The constant inner product is because . The inner product with vanishes because both first and third moments vanish. The squared norm is . Thus the relevant moments are the first through fourth, together with .
Quick check
(A fixed equal-weight design has , three orthonormal continuous basis functions, and Gram eigenvalues . If the target belongs to their span, find its expected integrated fitting error under independent noise variance .)
Solution · Answer
The approximation and alias terms vanish because the target is in the fixed space. The inverse-eigenvalue sum is , so the expected integrated error is . Normality is unnecessary; independent zero-mean errors with the stated finite variance suffice.
Quick check
(For the three nodes of the quadrature worked example, take the noiseless target . What is its degree-two sampled fit? Compute its continuous squared error and explain the zero training residual.)
Solution · Answer
All three nodes are roots of , so the sampled values are zero and the unique degree-two sampled fit is zero. Direct integration gives . The target is not determined by three point values within an unrestricted function family. Zero training residual therefore coexists with positive integrated error, even though the quadrature preserves degree-two basis products exactly.
Quick check
(Fit a constant to two measurements with independent equal noise variance using weights and . Find the variance of the fitted constant and its minimizing weight. Why can the Gram matrix alone not reveal this variance difference?)
Solution · Answer
The fitted constant is the weighted average, with variance . The minimum occurs at equal weights and is . With the constant basis, for every choice of weights, but the covariance also contains squared weights. Ignoring that extra matrix would erase the statistical distinction.