Earlier notes used Gaussian kernels as smoothing devices, for example in mean-shift, where points climb the gradient of a smoothed surface. This note studies that surface as a statistical estimator of an unknown density. We ask three questions. Is the estimate a genuine density? What are its exact mean and variance for a fixed sample size? How does the bandwidth trade bias against variance?
Setting and standing assumptions
Let be independent and identically distributed with Lebesgue density on , with . Fix a bandwidth that does not depend on the data.
Definition 7.8.1
(Kernel and kernel density estimator)
A kernel is a Borel-measurable function with and . For a bandwidth , the kernel density estimator is
Each observation contributes a bump of total mass centered at . The bandwidth sets the width of the bump.
For the pointwise moments below to be finite we assume:
- (A1) is bounded, say ;
- (A2) .
A bounded kernel satisfies (A2) automatically, because .
Common mistake
(Reading a density height as a probability)
- A density value is not a probability. For a continuous , . The value estimates a rate: roughly, for small .
- Density heights need not be at most . The uniform density on equals there.
- Heights carry units. If is recorded in meters rather than kilometers, every density value is divided by , and must be rescaled to match.
The estimate is a probability density
Theorem 7.8.2
(Sample-wise normalization)
For every realized sample and every , the function is nonnegative and .
Proof
(Finite sum and change of variables)
Nonnegativity holds because each term is nonnegative. For the integral, fix and substitute , so that :
The sum is finite, so the integral passes through it, and .
No probability was used. The statement is deterministic, so it holds even when is chosen from the data, since each realized is still positive.
Concept lensAlgorithmic
The estimate as a recipe for drawing a point
For a realized sample, is exactly the density of , where is uniform on and is independent of : pick an observation at random, then jitter it by times a kernel draw. Conditioning on gives the density , and averaging over gives . Normalization and nonnegativity are then automatic, and the same description says how to simulate from the estimate.
Exact finite-sample mean and variance
Theorem 7.8.3
(Exact pointwise moments)
Under (A1)–(A2), for every , every and every fixed ,
Proof
(One summand, then independence)
Write . These are iid because the are iid and is fixed.
Mean of one term. Substitute , so and the limits reverse:
Second moment. The same substitution gives
The average. Expanding the variance of a sum,
Independence makes every covariance zero, so the variance equals . Substituting completes the proof.
The mean is a smoothed version of . Unless is unchanged by averaging against , it differs from , and that difference is the bias. The variance is the sampling variability of the random number across hypothetical repeated samples. It is not .
The formula also has limits. With dependent or clustered data covariance terms can survive and the formula need not hold, although the mean formula still holds when the observations share the marginal .
Common mistake
(Using the fixed-bandwidth moments after choosing from the data)
If is chosen from the same sample, the terms in can become dependent, so the fixed-bandwidth variance formula is not generally valid without joint analysis of and the data. The normalization theorem does survive a data-chosen bandwidth, because it holds for every realized sample. The moment formulas do not, because they average over samples with held fixed.
Worked example 7.8.4
(Two observations, Gaussian kernel)
Take the sample , , and , where . Using , and :
The midpoint , where no data lie, receives a larger value than the observed point . Both bumps reach it equally, while at the far bump contributes little. With smaller the order reverses. Bandwidth controls shape, including the number of modes that mean-shift would find.
Worked example 7.8.5
(Box kernel and uniform data: interior point and edge)
Let and , so , and take .
Interior point . The window lies inside , so with ,
The estimator is exactly unbiased. The leading variance term omits the subtracted .
Edge . Only half the window meets the support: , so
Numbers. With and , the interior variance is with zero bias, while at the edge the variance is and the bias relative to is for every in the range. Changing within this range leaves both biases unchanged and moves only the variances: the edge bias is a property of the estimator, not of the bandwidth.
Bias in the smooth interior
Theorem 7.8.6
(Second-order bias)
Assume (A1). Assume also that is symmetric, , with . Finally, assume is twice differentiable on with , and is continuous at . Then, as ,
Proof
(Taylor expansion under the integral)
First moment of the kernel. Since , we have . Symmetry then gives .
Remainder term. Define
By Taylor's theorem in Lagrange form, there is between and with
Domination and pointwise limit. From this form, . For fixed , as , and continuity of at gives . Since is integrable, dominated convergence yields .
Assembling the bias. Integrating the definition of against , and using and ,
and the last term is .
Proof X-ray
What each hypothesis does in the bias expansion
The proof has one idea: expand to second order under the integral and let the moments of do the bookkeeping. Each hypothesis has one job. makes the zeroth-order term exactly . Symmetry of kills the first-order term, and without it the bias would be of order , namely . The finite second moment supplies the coefficient and, through , the integrability of .
The subtle step is the passage to the limit. Continuity of at gives only the pointwise limit ; it is the global bound that dominates the remainder by for every , including the large where is far from . With a merely local hypothesis the far tails of could see a region where is not smooth, which is exactly what happens at the boundary of a uniform density.
The sign of the bias is informative. Where , the leading bias is negative; where , it is positive. A local extremum can be flat, so its second derivative need not have a strict sign. If , the leading term vanishes, the bias is of smaller order, and the analysis below must be redone.
Variance asymptotics and consistency
Theorem 7.8.7
(Variance rate and mean-square consistency)
Assume (A1), (A2), and that is continuous at . Then as , whatever the accompanying values of . If and , then .
Proof
(Dominated convergence twice)
Variance. By the exact formula,
This expression does not involve . In the first integral, the integrand is dominated by and converges pointwise to , so the integral tends to . The second term is at most .
Bias. The bias is . Its integrand is dominated by and tends to pointwise, so the bias tends to .
Mean-square error. By the variance limit, . The MSE equals , so it tends to .
When we may write . When the theorem gives only , and the asymptotic-equivalence statement would be meaningless.
The leading-term bandwidth
Take , , and the hypotheses of both theorems. Keeping only the leading terms gives the approximate MSE
Write and . Setting the derivative to zero gives
The second derivative is positive, so this is the unique minimizer of the AMSE on . At the variance term equals , four times the squared-bias term, and , which is of order .
The trade-off is visible in the formula. Decreasing shrinks the smoothing bias but inflates the variance, which grows like . Increasing lowers the variance at any fixed , so the balance point moves toward smaller .
This is a balance of leading terms only. It depends on the unknown and . It is pointwise, whereas integrating over produces a different constant. It is not the minimizer of the exact finite-sample MSE. It changes form when or near boundaries. It carries no global-optimality guarantee.
Worked example 7.8.8
(Normal target: an oracle calculation)
Let , , and . Then and . Since ,
This is an oracle value: it uses knowledge of that no analyst has. It is not a selected bandwidth.
Exact bias. The exact moments are computable here. The mean is the density at , approximately . The exact bias is therefore about , while the leading term gives about .
Exact variance. We need . Subtracting the squared mean and dividing by gives
The leading term predicts .
Comparison. The exact MSE is about , against an AMSE of about . At the subtracted square is far from negligible. Asymptotics describe rates; the exact formula describes this sample size.
A boundary counterexample
Counterexample mode
Uniform data at the edge of the support
Let , and . Writing for the standard normal distribution function,
The right-hand limit of the true density at the boundary is . The value "" itself is arbitrary, because altering a density at one point leaves the distribution unchanged. The mismatch is therefore stated against the limit .
Half of the kernel window lies where no data can fall, so the estimator averages the left limit with the right limit . Smoothing also places mass outside . The bias theorem does not apply here: is not twice differentiable at , so its global hypothesis fails.
Interior points behave differently. At an exact calculation gives
which is exponentially small. This is consistent with the local zero-curvature intuition, but the stated global smoothness hypothesis still fails for the uniform density.
Boundary corrections. Reflection, for , is one remedy. Here it has mean at . Reflection and renormalization help only under explicit assumptions about the boundary and the local shape of . Neither guarantees unbiasedness in general.
Choosing from data
In practice is chosen empirically. A sound baseline uses held-out data:
- Split the sample into training and validation parts.
- Fit on the training part for each candidate .
- Score each candidate by the validation log-likelihood .
Any preprocessing, such as centering or rescaling, must be estimated on the training part only and then applied unchanged to the validation part. Otherwise validation information leaks into the fit.
The validation score is itself noisy and provides no automatic calibration. Under temporal or cluster dependence, random splits can leak information, and consistency results for iid data do not transfer automatically.
A density estimate describes how measurements are distributed. It supports no causal claim about what would happen under an intervention.
Exercises
Exercise 7.8.9
(Let for and otherwise, using the closed endpoints as written. With sample and , compute and .)
Foundation
Solution(3)
- 1
Evaluate at
- 2
Evaluate at
- 3
The endpoint convention
The point lies on the closed edge of both windows. With open endpoints, would be . A pointwise value at a kernel discontinuity depends on convention, while no integral does.
Exercise 7.8.10
(For the box kernel, find and . Verify directly that for the sample of the previous exercise, and explain why this holds for every sample and every .)
Foundation
Solution(3)
- 1
Kernel constants
- 2
Direct check
Here , and the overlap at is a single point. Hence .
- 3
The general reason
Substituting gives for each . Averaging ones gives , independently of the data and of .
Exercise 7.8.11
(Show that for iid terms the variance of the mean is . Use this to derive the exact variance of , and explain why the result can fail when .)
Core
Solution(3)
- 1
Variance of an average
Because: Independence removes the covariances; identical distribution makes the variances equal.
- 2
Moments of one summand
With , the substitution gives
and subtracting the squared mean yields the stated formula.
- 3
A data-dependent bandwidth
If depends on the sample, the terms all share a random, data-dependent bandwidth. They can be dependent and need not be distributed like the fixed- terms. The derivation therefore need not apply; a joint analysis of is needed.
Exercise 7.8.12
(For with a Gaussian kernel, derive . Explain why the density at is not intrinsically defined. Can setting , or choosing another , repair the bias relative to ?)
Core
Solution(3)
- 1
The mean at the edge
The same integral as in the counterexample gives .
- 2
What the density at means
Densities are determined only up to sets of Lebesgue measure zero. Any value assigned at gives the same distribution, so is not intrinsically defined. The intrinsic boundary quantity is the right-hand limit .
- 3
Neither repair works
Setting alters one point, changes no integral and moves the estimate further from . Choosing another does not help either: since for every , the mean stays below , and the bias relative to , namely , is below for every bandwidth and worsens as grows. For example, the mean is about at and about at . Only a change of estimator, such as reflection under stated assumptions, addresses the defect.
Summary
The kernel density estimate is a density for every sample. Its mean is a kernel-smoothed , and its exact variance for fixed and iid data is . In the smooth interior, the bias is approximately and the variance approximately . Their leading-term balance gives an oracle , not a universal prescription.
At support boundaries, at points where , under data-dependent bandwidths, and under dependent sampling, each of these conclusions must be re-derived.