Motivation
A useful image feature should be reproducible after a modest change of viewpoint or sampling. High brightness is insufficient: a bright uniform wall has little local position information. A strong edge is also incomplete, because moving along the edge may barely change its appearance. A detector therefore needs to measure how a patch responds to displacement in several directions, while choosing the spatial scale at which that response is meaningful.
There are two different averaging operations in this task. Derivative-scale smoothing controls which fluctuations contribute to a gradient. Integration-scale smoothing collects gradients from a surrounding patch. Combining them without distinguishing their roles makes a corner score difficult to interpret. A small integration window may see only one edge of a corner, while an excessively large window may combine unrelated objects.
This unit derives one complete mathematical detection pipeline. Its continuous formulas have exact transformation properties, while its discrete implementation has interpolation, boundary, and aliasing limitations. Keeping those two levels separate is essential: a rotation-covariant differential expression does not make a coarse pixel grid invariant under every rotation.
Definitions
Definition 5.1.1
(Gaussian smoothing and the structure tensor)
Let be a bounded image on . Write for Gaussian smoothing with standard deviation . Define and the integration-scale structure tensor
Its discrete version replaces the integral by nonnegative weighted samples, with a documented derivative and boundary convention. Products of derivatives are averaged; differentiating an already averaged outer product is a different operation. Since is a weighted sum of squared gradient projections, is positive semidefinite.
Definition 5.1.2
(Harris-type response)
For , define the Harris-type response .
It avoids explicit eigenvalue calculation while penalizing strongly unbalanced directional information. A detection algorithm first smooths and differentiates, forms and averages the three distinct tensor entries, computes , rejects scores below a positive threshold, and applies spatial nonmaximum suppression. A tie policy, such as a fixed lexicographic preference within a plateau, is needed to make the output reproducible.
Concept lensAlgebraic
The response as a condition on the eigenvalue ratio
Write the eigenvalues of as and put . Then
The sign of depends only on : exactly when lies below the larger root of . The condition is what keeps the balanced case positive. For , . The magnitude, by contrast, scales with , which is why a threshold on also tests strength.
Definition 5.1.3
(Difference of Gaussians and normalized Laplacian)
For scale selection define with , and . The first is a difference of Gaussians; the second is a scale-normalized Laplacian response.
A multiscale detector compares nearby spatial positions and nearby scales, rather than taking maxima independently at every scale and calling all of them distinct features.
Definition 5.1.4
(Gaussian pyramid)
A Gaussian pyramid stores smoothed images at successively coarser spatial grids.
If an existing level has effective standard deviation and a desired level has in the same coordinate units, the incremental Gaussian standard deviation is . This composition is exact for continuous Gaussian convolution and approximate for sampled or truncated kernels. After downsampling, scale and displacement values must be converted into the new pixel units.
Theorem/proposition
Theorem 5.1.5
(Local displacement information and rotation covariance)
For a twice continuously differentiable image on a patch, nonnegative finite weights , and sufficiently small displacement , the patch discrepancy satisfies
For the continuous isotropic Gaussian tensor, rotating the image by an orthogonal matrix gives at matching scales. Thus its eigenvalues, trace, determinant, and response are unchanged at corresponding points.
Theorem 5.1.6
(Difference of Gaussians and normalized scale derivatives)
For bounded , Gaussian smoothing satisfies for . Consequently
At fixed positive , as ,
This is an approximation limit, not exact equality for a finite scale ratio.
The tensor tells us which directions are locally distinguishable under a brightness model. It does not verify that a detected point corresponds to the same physical surface in another image. Likewise, a scale extremum selects a characteristic scale of a response, not a universal physical object size. The interpretation depends on the chosen image model.
Proof sketch or proof idea
Proof
(Taylor expansion and an orthogonal change of variables)
Taylor expansion gives . Squaring produces its quadratic projection plus a cubic remainder; finite weighted summation preserves that order. The smallest tensor eigenvalue is therefore the smallest leading discrepancy among unit displacement directions. An edge with all gradients parallel has a zero leading discrepancy in the tangential direction.
For rotation, isotropy gives and an orthogonal coordinate change preserves area. Thus the rotated smoothed image is and its gradient is . Substitute this into the tensor integral and change integration variable by . This yields orthogonal similarity. The exact statement uses a rotated continuous window; resampling a finite square grid introduces another approximation.
Proof X-ray
What invariance statement was actually proved?
Two hypotheses carry the displacement expansion. Twice continuous differentiability on a compact patch gives a uniform Taylor remainder at every sample, and a finite weighted sum keeps the remainder of order after squaring. The leading discrepancy in a unit direction is the Rayleigh quotient , which is smallest along an eigenvector of the smaller eigenvalue. That is why the smaller eigenvalue, not the trace, measures the worst direction.
The rotation part uses isotropy twice: of the derivative Gaussian and of the window . With a square box window instead, the same change of variables works only for rotations that map the box to itself, such as quarter turns. What was proved is covariance under exact continuous rotation, not contrast invariance. Under , gradients multiply by , tensors by , and the response by , so a fixed numerical threshold can change decisions while the geometry is unchanged.
The scale relation needs a separate and shorter computation.
Proof
(Differentiating the Gaussian in scale)
To obtain the scale relation, differentiate the Gaussian formula directly. In two dimensions both and equal . Gaussian derivative bounds permit differentiation under convolution for bounded at positive scale. Integrating in scale gives the identity, and continuity of the integrand gives the limit after division by . The normalization factor is therefore derived from changing scale, rather than inserted merely to improve the appearance of a plot.
For spatial scaling with , substitution in the Gaussian integral gives . Scale-normalized gradients and the normalized Laplacian then agree at corresponding scaled locations and scales. Both derivative and integration scales must be multiplied by . Using an unchanged window after resizing an image is not covered by this calculation, and quantized scale levels generally locate the ideal extremum only approximately.
Worked examples
Worked example 5.1.7
(A corner score distinguishes directional balance)
With , an edge tensor has response . A tensor has response . Doubling image contrast multiplies the latter tensor by four and its response by sixteen, giving . A fixed threshold between the two responses would change detections under contrast scaling, despite identical patch geometry. A large positive response requires adequate strength and balance; rank alone does not say whether either is above the noise floor.
Worked example 5.1.8
(An ideal Gaussian blob selects its own scale)
Let with . Completing the Gaussian convolution gives . At the center,
Write . Its magnitude is , whose derivative is . The unique positive maximum occurs at , or , and equals one half. This result concerns this specified amplitude and profile; an arbitrary object need not have a Gaussian blob's scale interpretation.
Worked example 5.1.9
(Downsampling changes which frequency is seen)
Consider sampled data . Taking every second sample gives , a lower-frequency sequence on the coarse grid. The original frequency cannot be inferred uniquely from that sequence. Gaussian prefiltering attenuates the high-frequency component before sampling, but a Gaussian is not exactly band-limiting: its Fourier multiplier is positive at every finite frequency. An exact no-alias theorem requires a bandlimit assumption; practical smoothing reduces aliasing rather than proving its complete absence.
The tensor and scale responses can be combined without conflating their roles. At each candidate scale, compute directional information with a corresponding integration window, reject poorly conditioned patches, and then compare an appropriate scale response across adjacent levels. For a difference-of-Gaussians detector, examine a three-by-three spatial neighborhood at the current and adjacent scales; the central sample has twenty-six neighbors. Boundary samples lack a complete neighborhood and need an explicit exclusion or extension policy.
Localization can be refined by fitting a quadratic to neighboring response values. If its Hessian is invertible, the stationary offset solves a small linear system. A nearly singular Hessian means the fitted location is sensitive; an offset outside the fitted neighborhood should not be accepted as a trustworthy local interpolation. Refinement is useful only when the discrete samples reasonably resolve the response. It cannot create detail already removed by averaging or restore an aliased frequency.
To make this procedure reproducible, keep two coordinate descriptions for each candidate: its pixel location in the current pyramid level and its position in original-image units. A location at pixel ten on a grid of spacing four corresponds to forty original pixel spacings from the same origin. Offsets introduced by padding or filter centering must also be retained. When comparing detections between images, transform positions, scales, and any assigned orientations consistently. A score table without these geometric conventions cannot determine a correspondence. Even with them, descriptor comparison and outlier rejection are additional tasks: this detector supplies candidates and local conditioning information, rather than a complete matching certificate.
Common mistakes
Most errors here come from reading one desirable property of a detector as if it implied another.
Counterexample mode
A balanced tensor is not necessarily a strong feature
Take with arbitrarily small. Its condition number is one, yet its response is , tending to zero. Balance alone does not supply reliable localization. Conversely, a huge single eigenvalue can coexist with a zero second eigenvalue and leave a motion direction completely ambiguous. Strength and directional balance must both be assessed.
Common mistake
(Adding blur standard deviations)
To go from effective blur to effective blur , convolving with a Gaussian of standard deviation gives total blur , not . Variances add under Gaussian convolution, so the required increment is .
Common mistake
(Treating one covariance property as robustness)
Finite differences can respond strongly to isolated noise; using a large integration window does not undo a poorly chosen derivative scale. Nonmaximum suppression should follow a defined score and threshold, not replace them. Plateaus require tie handling and may reflect unresolved localization. Finally, rotation covariance, scale covariance, contrast normalization, and robustness to viewpoint change are separate properties. Proving one does not prove the others or establish reliable feature matching.
Summary
Patch displacement leads to the structure tensor and a directional corner response. Gaussian derivative identities relate nearby-scale differences to the normalized Laplacian, and a blob model makes scale selection explicit. A practical pyramid must track effective blur, coordinate units, neighborhoods, and boundaries. Continuous covariance supplies a useful ideal, while sampling, noise, contrast thresholds, and localization conditioning limit what a finite detector can guarantee.
Exercises
Exercise 5.1.10
(For and , compute . Rotate the tensor by ninety degrees and verify the same response.)
Foundation
Solution(2)
- 1
Compute the response
The determinant is four and the trace five, so .
- 2
Rotate and recompute
A ninety-degree rotation swaps the diagonal entries, giving the same determinant and trace and hence . General orthogonal similarity preserves both through the characteristic polynomial; this is one explicit instance of the theorem.
Exercise 5.1.11
(An image level has effective blur two pixels, and the desired next level has blur three pixels in the same coordinates. Find the incremental blur. After downsampling by two, express the final blur in coarse pixel units.)
Foundation
Solution(2)
- 1
Subtract variances
The variance increment is , so the incremental standard deviation is .
Because: Variances, not standard deviations, add under Gaussian convolution.
- 2
Convert units
The resulting blur is three fine pixels, or coarse pixels after the grid spacing doubles.
Exercise 5.1.12
(For the Gaussian blob example, evaluate the normalized response magnitude at and . Explain the equal values without asserting two maxima.)
Core
Solution(2)
- 1
Evaluate the two scales
The corresponding values are and , and both give .
- 2
Use the symmetry and monotonicity
The function is unchanged by . It increases up to and then decreases, so equal values on opposite sides do not create additional maxima.
Exercise 5.1.13
(Show that adding a constant brightness offset does not change the tensor. Explain why multiplying intensity by a nonzero constant still requires threshold adjustment, and why neither transformation proves robustness to spatially varying illumination.)
Core
Solution(3)
- 1
Add a constant
The gradient of an added constant is zero, so every outer product, and hence the tensor, is unchanged.
- 2
Multiply by a constant
Multiplication by yields response , which requires a corresponding threshold change to preserve decisions.
- 3
Let the change vary in space
For spatially varying or , the product rule introduces extra gradients. Those can alter the tensor and create apparent features, so the constant-illumination derivation no longer applies.