Motivation
A blurred measurement may fit many possible images. Dividing by a small blur coefficient can reproduce the measurement closely, yet greatly amplify noise. To choose a reconstruction we need more than the observed residual: we need a statement about which errors matter and what is known about the signal and noise. Here the image and the measurement are finite random vectors. The resulting filter is optimal within a precisely specified class, not automatically for every photograph.
Suppose , where is the unknown image, is a known real matrix, and is additive noise. We seek an affine estimate minimizing the expected squared error . Expectations require finite second moments. This criterion averages over the assumed distribution; it does not guarantee the smallest error for each individual image.
Definitions
Write , , and . Assume is positive definite. The error of an estimate is ; its risk is its expected squared norm. In general, signal and noise may be correlated; independence is not needed for the next theorem. The simpler formulas after it explicitly assume they are uncorrelated.
Theorem/proposition
Theorem 1.6.1
(Best affine estimator)
If and have finite second moments and is positive definite, the unique affine estimator as a random vector that minimizes risk is
Its error is uncorrelated with every centered linear function of . Among all affine estimates, the excess risk over this estimate is after optimizing the intercept, where .
Proof sketch or proof idea
Proof
(Centering the intercept, then an orthogonality identity)
For a fixed , write . The estimation error is , where has mean zero, so
Thus the best intercept has . Put . Direct substitution gives . For another matrix , the centered error is , and expanding its square gives
The cross term vanishes by the covariance identity, leaving the displayed nonnegative trace. Positive definiteness makes the trace zero only when . This is an orthogonal projection in the space of square-integrable random variables, but only onto the affine functions of the observation.
Proof X-ray
Where the assumptions enter
The proof makes two separate reductions. The first is a bias–variance split: an intercept only shifts the mean of the error, so it can be optimized independently of , and the optimal choice makes the error centered. The second is Pythagoras: the identity says the error of is orthogonal to every centered linear function of , so any other matrix adds a term and can remove none.
Finite second moments are what make every expectation and cross term finite. Positive definiteness of is used twice: to define through the inverse, and to make the excess trace strictly positive. If is singular, an optimum can still exist, but its matrix representation need not be unique; the displayed inverse cannot be used unchanged. Nothing about independence or Gaussianity is used, which is exactly why the conclusion is restricted to affine estimators; orthogonality of the error to the observations does not assert statistical independence.
If and are uncorrelated, and . Thus known second moments determine the optimal affine estimate. In diagonal form this gives the familiar scalar filter.
Corollary 1.6.2
(Scalar gains for uncorrelated noise)
Consider zero-mean independent scalar coordinates , with and uncorrelated, signal variance and noise variance , and assume . The optimal affine estimate and its risk are
Proof
(Substituting the scalar moments)
In each coordinate and , so the theorem gives the gain . Its risk is .
This applies, for example, in a real orthonormal basis jointly diagonalizing a symmetric blur and the two covariances. A periodic convolution may instead use complex Fourier coefficients, in which case transpose becomes conjugate transpose and the numerator becomes . This diagonal formula requires the stated covariance structure; stationarity or diagonalization must not simply be assumed for arbitrary images.
Worked examples
Worked example 1.6.3
(A weakly observed mode)
Let , , and . Then . The unregularized inverse gain is . The optimal affine risk is
For the inverse estimate, the risk is . A zero estimate has risk . The gain of eight balances noise and attenuation under this model; it does not recover every realization exactly.
Worked example 1.6.4
(Connection to a quadratic penalty)
For , minimizing gives . Thus this particular quadratic penalty and the linear minimum-risk estimate agree when their parameters have the stated variance ratio. A generic penalty parameter is not automatically a known noise-to-signal variance ratio. For independent jointly Gaussian , the affine estimator is also the conditional mean and hence minimizes risk among all measurable estimators with finite risk; the affine theorem above does not require Gaussianity.
Worked example 1.6.5
(A noise-free mode turns the filter into the inverse)
Keep and but set . The condition holds, and
Without noise, exact inversion is optimal and recovers from . Regularization in the first example was therefore caused by , not by the small value of alone. The opposite boundary case, , is treated in the quick check below.
Common mistakes
Counterexample mode
A smaller residual need not mean a smaller risk
In the first example the inverse estimate has zero residual , but its risk is , larger than for gain . The latter residual is . Fitting the noisy observation exactly also fits its noise, so residual and expected reconstruction error answer different questions.
The theorem's restriction to affine estimators is also easy to forget.
Common mistake
(Calling the affine optimum optimal among all estimators)
Uncorrelatedness is not independence, and the optimal affine estimator need not be the optimal nonlinear one. For example, take equally likely to be or , and independent equally likely to be or . The four possible observations uniquely determine , so a nonlinear lookup has zero risk. Yet the best linear gain is and has risk . Gaussian assumptions cannot be omitted when claiming conditional-mean linearity.
Quick check
Quick check
(A mode has , and . What are its optimal gain and risk in the uncorrelated model?)
Use the scalar gain formula.
Solution · Answer
The gain is , so the best affine estimate of that centered mode is zero and the risk remains . The measurement contains no linear predictive information about the mode in this uncorrelated model. Nonlinear information may nevertheless be present.
Exercises
Exercise 1.6.6
(For , , , find the optimal gain and risk.)
Foundation
Solution(2)
- 1
Compute the gain
.
- 2
Compute the risk
.
Exercise 1.6.7
(With , , , compare the risks of gains , , and .)
Foundation
Solution(2)
- 1
Write the risk of a gain
The error is , with uncorrelated terms, so the risk is .
- 2
Evaluate
It equals , and respectively. The middle value is the optimal gain .
Exercise 1.6.8
(Let be a centered scalar with variance one and . Compute the correct optimal gain; explain why the uncorrelated-noise formula fails.)
Core
Solution(3)
- 1
Compute the true moments
Here , so and .
- 2
Apply the theorem
The gain is , and : the error is zero.
- 3
Compare with the simplified formula
The simplified formula omits the nonzero cross-covariance of signal and noise: it would use and . Their ratio happens to give the same gain here, but the formula's risk is wrong, since the true risk is zero.
Exercise 1.6.9
(If is positive definite, show that changing by a nonzero matrix strictly increases the optimally centered risk.)
CoreProof
Solution(3)
- 1
Use the excess-risk formula
The excess risk of is .
- 2
Factor the covariance
Key step
The trace equals .
Because: A positive definite matrix has an invertible positive definite square root.
- 3
Conclude
∎If , then because is invertible, so the excess risk is strictly positive.