Motivation
Placing a person against a new background seems like a cutting problem until we inspect hair, glass or motion blur. At such pixels, the measured color contains contributions from more than one surface. A hard foreground mask cannot describe that mixture. Conversely, a soft mask alone does not tell us the foreground color: the original background may already be mixed into the stored pixel. Compositing combines known layers; matting tries to recover the layers that produced an observation. Their equations point in opposite directions and have very different information requirements.
A panorama raises a related issue. Two photographs may agree geometrically yet disagree in brightness, contain moving people, or view different sides of an object. Averaging their overlap can hide a small brightness discontinuity, but it can also create a translucent double person. We therefore develop a sequence of distinct modeling decisions: coordinate alignment, photometric agreement, visibility selection and blending. Treating all four as “make the seam disappear” conceals the reason a method succeeds or fails.
The calculations below use linear light RGB values. A componentwise average of display-encoded colors generally does not represent the same physical mixture. We also assume the layers have already been expressed in the same color space and coordinate system. Neither assumption is guaranteed merely because the files use three channels and have identical dimensions.
Definitions
Definition 1.2.1
(The matting equation)
The basic matting equation at one pixel is
where are observed, foreground and background linear colors. The scalar may describe fractional coverage in a simple pixel model. Its interpretation as optical opacity requires additional assumptions about transmission and color; colored glass need not be represented accurately by one scalar alpha. The equation is a model, not a universal description of every optical interaction.
Definition 1.2.2
(Premultiplied layers and the over operation)
An unassociated or straight layer stores . A premultiplied layer stores with . At , a bounded foreground color has premultiplied color zero, even though its unassociated color is irrelevant to the visible composite. For premultiplied layers and , define the ordered “over” operation by
The result is another premultiplied layer. A final opaque background has alpha one. The rule specifies how the front layer attenuates the contribution of everything behind it. For coverage interpretations involving correlated subpixel masks, the simple product of uncovered fractions may require a more detailed visibility model; the algebraic operator itself is still well defined.
Definition 1.2.3
(Feathered blend and seam selection)
A mosaic uses warped source images on a common coordinate domain. Each image has a validity mask . A feathered blend has weights , with zero weight outside valid regions, and
where the denominator is positive. The normalization preserves a constant image. A seam instead selects which source contributes in a region before a transition is blended. Selection and averaging solve different problems, especially when observations disagree because something moved.
Theorem/proposition
Theorem 1.2.4
(Premultiplied over is associative but ordered)
For arbitrary premultiplied layers with alphas in ,
Their common alpha is . The operation is generally not commutative: changing front-to-back order can change color even when it leaves alpha unchanged.
Theorem 1.2.5
(Two known backgrounds can identify a matte)
Suppose two noise-free captures share the same foreground and alpha , are registered, and use known distinct linear background colors :
Then
If , the foreground is uniquely . At , alpha is identifiable but foreground color is not. Data inconsistent with the model need not yield or a zero residual.
This result explains the value of changing the background deliberately. It does not promise that two ordinary photographs suffice: foreground motion, illumination changes or a different camera response can violate the shared-foreground assumptions. Distinct backgrounds with almost identical colors also lead to a small denominator and poor noise sensitivity. Identifiability in exact arithmetic is weaker than stable estimation.
Proof sketch or proof idea
Proof
(Transmittance bookkeeping for three layers)
Write for the front layer's transmittance. Composing two layers gives color and transmittance . Either grouping of three layers therefore gives
Associativity follows from distributivity and associativity of scalar multiplication. The calculation also explains why a group of layers can be cached as one premultiplied layer without changing its interaction with a later background. It does not permit rearranging the layers: the coefficient attached to each color depends on which layers lie in front.
Proof
(Subtracting the two captures)
For the matting theorem, subtract the two observation equations. The unknown foreground term cancels, leaving . Taking its inner product with the known nonzero background difference yields the formula. Substitution recovers only if division by alpha is possible. In noisy data the same formula minimizes over real ; constraining clips that scalar minimizer to the interval. A residual perpendicular to the background difference diagnoses disagreement not explained by this model.
Proof X-ray
Counting unknowns is a warning; cancellation supplies the solution
One RGB observation gives three scalar equations for seven scalar unknowns: three foreground channels, three background channels and alpha. Even a known background leaves four unknowns. Smoothness, a trimap or learned statistics provide additional assumptions; they do not follow from the pixel equation. The second controlled background is useful because subtraction eliminates the three foreground unknowns at once. A successful inverse problem needs such independent information, not simply a more elaborate optimizer.
Photometric matching in a mosaic offers another gauge issue. Suppose corresponding positive intensities satisfy , where is an image exposure gain and is the common scene value. Log differences constrain , with . Adding the same constant to every leaves every difference unchanged. On a connected overlap graph, fixing one gain removes this ambiguity: if two exact solutions agree at the fixed image, their difference is constant along every edge and hence zero everywhere. Disconnected image groups require separate anchors or additional observations.
Worked examples
Worked example 1.2.6
(Layer order and premultiplication)
Let a half-opaque red layer be and a half-opaque blue layer be . Red over blue gives , while blue over red gives . Place either result over opaque black: its premultiplied color is already the final visible color. Dividing by would instead recover the group's unassociated foreground color, which is a different quantity. Alpha agrees in both orders because uncovered fractions multiply, but the visible colors differ.
Worked example 1.2.7
(Recovering a controlled-background matte)
Take , , and observations , . Their difference is , exactly half the background difference. Thus and . The foreground is . Against a new blue background , the predicted color is . Simply copying onto that background would lose the background contribution and misrepresent the edge.
Worked example 1.2.8
(A feathered overlap that creates a ghost)
At one aligned location, image one contains a bright moving object with scalar intensity one; image two shows the dark background with intensity zero. Equal feather weights produce . That value is neither observed object appearance nor observed background appearance. Changing weights smoothly makes the ghost fade gradually but does not establish which scene state should be depicted. A seam selecting one consistent observation can avoid this double exposure, provided it can pass around the object without creating a worse discontinuity elsewhere. Alignment cannot solve this temporal visibility decision.
Filtering a layer is another reason to store premultiplied color. Interpolating straight colors near transparency can mix an invisible arbitrary color into visible pixels. Interpolating and alpha together gives the correct weighted layer contribution under the adopted model. If an unassociated color is needed afterwards, divide only where alpha is positive; values near zero remain numerically sensitive and often have little visible relevance.
Concept lensStructural
A mosaic is a graph before it is one image
Each photograph is a vertex; a reliable overlap supplies an edge carrying geometric and photometric constraints. A chain of locally plausible pairwise transforms can still accumulate drift. A cycle supplies a consistency check: traversing the cycle should approximately return to the starting coordinate frame and exposure. Joint adjustment uses redundant edges to distribute disagreement. A blend operates after these constraints and cannot certify their consistency.
A useful practical order is to estimate transformations, inspect overlap residuals, compensate justified exposure differences, choose seams around inconsistent regions, and finally blend remaining transitions. In a multiscale blend, low-frequency differences can be spread over a wider region while high-frequency detail changes near a narrower seam. This can reduce visible transitions, but it is not a proof of geometric correctness. Strong parallax may require depth-aware warps or a different acquisition strategy.
Common mistakes
Counterexample mode
Knowing the background does not determine alpha
For one scalar channel, let the known background be zero and observe . Both and satisfy the equation. All values lie in . Thus valid color bounds and a known background still do not identify a unique matte from one observation. The repaired claim needs more information, such as a known foreground color, additional controlled captures or justified spatial constraints.
Common mistake
(Segmentation confidence is not alpha)
Do not confuse a segmentation probability with a physical mixing coefficient. A classifier's confidence that a pixel belongs to a person is a statement about uncertainty; alpha describes how a specified compositing model mixes contributions. They may be correlated, but substituting one for the other requires justification. A trimap similarly labels confident foreground, confident background and an uncertain region; it does not provide the unknown colors automatically.
Common mistake
(Feathering does not repair geometry or photometry)
Do not expect a feathered blend to repair lens distortion, missing coverage, a moving subject or a change in viewpoint that violates a planar model. Nor does matching global average brightness prove local photometric agreement: shadows and vignetting can vary spatially. Diagnose the residual pattern before adding parameters. An exposure model that fits every local discrepancy may be absorbing geometry or scene changes rather than describing camera gain.
Summary
Premultiplied compositing has an associative ordered algebra, which supports filtering and grouping layers consistently. Matting reverses a mixture equation and is underdetermined without additional information. Controlled background changes can supply that information, subject to visibility, registration and stability conditions. A coherent mosaic separates geometric alignment, photometric constraints, source selection and blending, and uses overlap consistency as evidence rather than relying only on a smooth-looking seam.
Exercises
Quick check
(Compose premultiplied scalar layers and . Find their color and alpha, then place the result over opaque white. Verify the final value by expanding the three-layer formula.)
Solution · Answer
The group color is and alpha is . Over white, the final intensity is . Expansion gives . This uses premultiplied inputs; multiplying the stored colors by alpha again would be an error.
Quick check
(Two scalar-background captures have , , and . Recover alpha and foreground. Predict the composite over background zero.)
Solution · Answer
The background difference is and the observation difference is , so and alpha is . The foreground is . Against zero the composite is . Substitution into the second capture gives , providing an independent arithmetic check.
Quick check
(A foreground is recovered by with known alpha and background. Show how an observation perturbation changes . Compare alpha and for .)
Solution · Answer
Holding alpha and background fixed gives . The magnitudes are and respectively. Thus a nearly transparent region can have a poorly determined unassociated foreground even when its visible premultiplied contribution is stable. If alpha is also estimated, its error introduces further terms and cannot be ignored.
Quick check
(Three images form a connected overlap chain. Measured exact log-gain differences are and . Set and find the gains. What difference should an additional edge report, and what does disagreement establish?)
Solution · Answer
We obtain and , hence gains . The closing edge should report . Disagreement proves that the exact shared-gain model and all edge measurements cannot simultaneously hold. It does not by itself identify which edge is wrong; noise, correspondence errors and nonuniform illumination are distinct possible causes.