Motivation
Many color cameras sample different spectral bands at different sensor locations. A reconstructed color image therefore contains both measurements and estimates. The earlier HDR unit studies intensity and exposure; this unit examines the spatially interleaved color measurements that can precede a color image. We use a small idealized model to see why a plausible reconstruction need not be uniquely determined by the sensor data.
One measurement per sensor site
Definition 4.1.1
(A periodic color-filter array)
At integer pixel coordinates , use a repeating arrangement with red at even-even sites, blue at odd-odd sites, and green at the other two sites. Suppose an ideal linear sensor records one scalar , where is the channel selected by that site's filter and is measurement error. Demosaicing estimates the two missing channels at each location.
The labels here are sensor-channel values, not automatically display-ready RGB. Real filters have overlapping spectral sensitivities; a more detailed linear model integrates incident spectral energy against the response of the selected filter. Exposure, dark offset, saturation, optical blur and sensor noise also matter. We assume offsets have been corrected, values are unsaturated, and channel values are linear for the calculations below. Changing this pattern, its coordinate origin or its boundary convention changes which neighbors are used.
Concept lensStructural
The mosaic as three masked channel images
Equivalently, let be the indicator images of the three site types, which are disjoint and cover every pixel, with densities , and . Then pixelwise. Demosaicing inverts a map that keeps each channel on only its own sites; any change to a channel supported off those sites leaves the data unchanged.
Local interpolation
Definition 4.1.2
(Bilinear estimate at an interior green site)
At a green site under the stated pattern, the red sites immediately left and right are and ; the blue sites immediately above and below are and . A simple interior bilinear estimate takes
These are modeling choices, not identities for arbitrary scenes. At an image boundary, where a listed neighbor is unavailable, an explicit alternative rule is required.
Proposition 4.1.3
(Bias of the midpoint average)
Let a channel along one coordinate be a quadratic , with noiseless samples at and . Then
In particular the average equals the true midpoint value for every affine channel (), and for no channel with .
Proof
(Expanding the two samples)
Compute . Adding the two values, the terms in cancel and
Dividing by gives the stated identity.
Proof X-ray
What averaging assumes
The only property of the sample positions used is symmetry about the target : it makes the odd terms in cancel, so the average sees the channel's constant and linear parts exactly. The leftover is the cost of curvature, which is why the rule is exact on local affine channels and wrong on anything bent. For a smooth channel the same cancellation leaves a bias proportional to the second derivative times .
Noiselessness is a separate hypothesis: with noisy samples the average also contains noise, treated below. A sharp edge or oscillatory texture need not be affine between the samples: the mean is still computable, but it is not necessarily the true missing value. The analogous statement holds vertically for blue.
Worked example 4.1.4
(Reconstructing one interior site)
At let the observed green value be , the red neighbors be and , and the vertical blue neighbors be and . The reconstructed sensor-channel triple is . For example, gains applied in linear space yield before any clipping or display encoding. Gains are a declared processing choice; this calculation does not by itself calibrate color to a display.
Worked example 4.1.5
(A red site needs diagonal and four-neighbor averages)
At the red site the four edge neighbors and are green, and the four diagonal neighbors are blue. The same bilinear idea averages them. Let the observed red be , the green neighbors be and the diagonal blue neighbors be . Then and , so the triple is .
Worked example 4.1.6
(A site on the image boundary)
Suppose the image occupies and . The site is green, with red neighbors and above and below and blue neighbors and left and right. The pixel does not exist, so the interior formula for blue cannot be applied. Take the stated policy of mirroring the missing neighbor about the edge, which replaces by . With , and , we get and . A different policy, such as zero padding, would give : the boundary rule is part of the algorithm.
What the samples cannot determine
Counterexample mode
Two different red images give identical samples
On the infinite integer grid, compare with , and let their green and blue channels be identical. At every red site is even, so the sine vanishes and the recorded red samples agree; green and blue samples agree by construction. Yet at the green site the true red values are and . Both are within . The complete ideal mosaics are identical, so no deterministic algorithm receiving only that mosaic can recover both original images exactly.
This is an identifiability limit, not a claim that natural photographs always have this pattern. Smoothness, edge models or learned statistics can supply preferences, but they are additional assumptions. Interpolating across an edge may mix values from different surfaces and create color artifacts. Comparing only reconstructed values to the samples used to create them cannot by itself measure accuracy at unobserved channels.
Common mistake
(Treating the reconstructed RGB triple as measured)
At every pixel two of the three reconstructed channels are estimates. In the first worked example the sensor measured only ; the values and come from neighbors, and the counterexample shows that they can be wrong without any inconsistency with the data. A reconstruction that reproduces the mosaic exactly is therefore not evidence that the full color image is right.
Noise and processing order
Proposition 4.1.7
(Variance of an average of two measurements)
Let be measurement errors with variances and covariance . Then
In particular, if are independent and zero-mean with common variance , the variance is .
Proof
(Bilinearity of the variance)
Variance is quadratic: , and multiplying a random variable by multiplies its variance by . For independent errors , and with this gives .
If the two red measurements used above have independent, zero-mean errors of common variance , the noise in their average therefore has variance . This is variance of the measurement noise under the stated model; it does not include interpolation bias from a non-affine scene. Neighboring reconstructed pixels may reuse a measured sample and therefore have correlated errors. For example, if all distinct sample errors are independent with common variance , two averages each using weight and sharing exactly one sample have covariance .
A white-balance gain multiplies both the linear channel and its noise; a gain multiplies noise variance by . Display encoding is generally nonlinear, so applying the same averaging arithmetic after encoding is not generally equivalent to averaging linear values first. The precise camera processing pipeline, clipping and noise model must be specified before treating any numerical result as a calibrated color measurement.
Common mistake
(Averaging after a nonlinear encoding)
Use the square root as a simple stand-in for a display encoding. Two linear neighbors and have linear mean , whose encoded value is . Encoding first gives and , and their average is . The two orders disagree, so interpolation belongs on linear values, and the encoding is applied afterwards.
Exercises
Quick check
(At a green site of the stated type, red neighbors are and , blue neighbors are and , and measured green is . Find the reconstructed triple.)
Average each pair of neighbors; green is measured.
Solution · Answer
The red estimate is , green is , and blue is , giving .
Quick check
(If the two red errors are independent with variance each, find the variance of their average. What changes if their covariance is ?)
Use .
Solution · Answer
Independent errors give . With covariance , the variance is .
Quick check
(Why can the two red images in the counterexample produce the same mosaic but different missing red values?)
Which sites does the filter array use to measure red?
Solution · Answer
Their difference vanishes at every red-filter site, while the filter array never measures red at the green site . An estimate at that site therefore needs an additional assumption.
Quick check
(At a boundary with a missing interpolation neighbor, does the displayed interior formula define a value? State what must be added.)
Look at which neighbors the formula lists.
Solution · Answer
No. A boundary extension, one-sided estimate or another specified rule is needed; each changes the effective model and must be assessed separately.
Further study
Edge-aware demosaicing, color calibration, noise covariance, spatially varying optical blur and joint HDR reconstruction need additional analysis. This chapter does not implement a production camera pipeline or guarantee color accuracy for a particular sensor.