Why a density curve needs two error accounts
A kernel estimate displays a smooth-looking curve, but a curve alone does not say how much it would change with another sample. Even infinitely many observations would converge, for a fixed bandwidth, to a smoothed version of the density rather than necessarily to the density itself. We therefore need two accounts: sampling error around the expected estimate, and smoothing bias between that expectation and the target density.
This note gives a finite-sample statement at finitely many declared locations. It uses a box kernel because its estimate is an ordinary count divided by a window volume. The resulting band is conservative, but all its assumptions and calculations are visible. It does not claim coverage at every point of a continuum, or validity when the assumed smoothness bound is invented from the same data.
Definitions and assumptions
Definition 18.1.1
(Box-kernel estimate and a smoothness class)
Let be independent, identically distributed real random variables with density . Fix and define
Assume that a known number satisfies for all real . This global condition is convenient; the proofs need it only on each window used. Let be positive integers. Fix locations and bandwidths before seeing the data. Let .
The density is defined up to changes on sets of measure zero; the Lipschitz assumption selects its continuous representative, which is what means here. The observations may be repeated as numerical values, but their random draws must be independent. For a density known to have bounded support, extend it by zero outside the support before checking the global assumption. A jump at a support boundary violates it.
Concept lensAnalytic
The estimate as a difference quotient of the empirical CDF
Let be the empirical distribution function and the true one. Then and , because . The estimate is a symmetric difference quotient of , and the bias measures how far that quotient of lies from its derivative.
Theorem 18.1.2
(Simultaneous finite-grid band)
Put
With probability at least , all inequalities
hold simultaneously. Thus is a simultaneous band on this finite location-bandwidth grid. The probability is over the sample; the true density and the declared grid are fixed.
Proof
(Separate random error from deterministic bias)
For one fixed pair , each summand lies in and the summands are independent. Hoeffding's inequality gives
At this is at most . A union bound over all fixed pairs bounds the probability of any sampling failure by ; independence between estimates is neither required nor asserted.
For the bias, average the Lipschitz inequality inside the window:
The triangle inequality combines the two bounds. Truncating the lower end at zero preserves coverage because a density is nonnegative.
Proof X-ray
What the probability statement protects
The random part is the deviation from the window average, not directly from the unsmoothed density. The bias bound is deterministic only when the smoothness bound is valid for the true density. A union bound buys simultaneous protection for a finite list, not automatically for uncountably many locations.
Bias, variance and choosing a bandwidth
Theorem 18.1.3
(An exact variance and a pointwise mean-square bound)
For a fixed , write . Then
Under the stated Lipschitz condition, , so the mean-square error is at most
Proof
(Binomial count and the mean-square decomposition)
Let . The indicators are independent Bernoulli variables with success probability , so is binomial with variance . Since , its variance is and its mean is . Write and square: the cross term has expectation , so the mean-square error is the squared bias plus the variance.
For the bound, , and the averaging step used for the bias bound gives , so . With the variance is at most , and the squared bias is at most .
Here the useful stochastic scale is roughly , while the distribution-free Hoeffding radius above has the more conservative scale . Both are valid in their own statements; substituting the variance scale into a confidence formula without a tail argument is not justified.
As decreases, shrinks while the displayed Hoeffding radius increases. If and are fixed, minimizing their sum over continuous positive gives
This optimizes a conservative upper bound, not the unknown actual error. If , the displayed expression divides by zero and the optimization must be reconsidered. In fact no probability density on the whole real line can be globally constant; a local condition on the relevant windows can still be useful.
A bandwidth may be chosen after examining the sample from the declared finite list: on the simultaneous event the inequality holds for every candidate, hence also for the selected candidate. The factor in the radius pays for this protection. A new candidate introduced after seeing the data is not covered by that finite-list argument. Alternatively, choose a bandwidth using an independent pilot sample, then condition on that choice while applying the fixed-bandwidth statement to fresh estimation observations; the conditional assumptions must still hold.
Worked examples
Worked example 18.1.4
(A count and a simultaneous uncertainty interval)
Suppose , one fixed location , one bandwidth , , and a valid bound . If observations fall in the window, the estimate is . The sampling radius is
The reported interval is approximately . The theorem says that this procedure covers the fixed true value at least 95% of the time under its assumptions. It does not say there is a 95% probability that this fixed value lies in this particular observed interval.
Worked example 18.1.5
(A window at a jump)
For the uniform density on and , the unreflected box estimate at zero has expectation . Its continuous interior value approaches one, so the half-height error does not vanish as . Extending the density by zero produces a jump and violates any finite global Lipschitz bound. A reflected estimator changes the bias calculation; the above formula must not be transferred to it without a new proof.
Worked example 18.1.6
(Choosing from a finite bandwidth list, with truncation at zero)
Take , , a single location, the list (so and ) and . Then , so , while .
The middle bandwidth gives the narrowest guaranteed band on this list. The continuous minimizer of the displayed formula, with the same , is with half-width about ; it is only a benchmark, because that bandwidth is not on the declared list. Since the guarantee is simultaneous over all three candidates, the analyst may look at the data and then report .
Now take a tail location where the window count at is , so the estimate is . The raw interval would be , but a density is nonnegative, so the reported band is : the lower end is truncated at zero and the upper end is unchanged.
Counterexample mode
Dependent copies do not provide n independent observations
Let be uniform on and put . At and , the estimate is either or , each with probability , regardless of . It cannot concentrate around its expectation merely because the same observation was copied many times. The iid hypothesis is a mathematical requirement, not a formatting choice for a dataset.
Reading the band responsibly
A band around accounts only for sampling. A band for additionally needs a valid bias bound. A claim about the entire real line needs control between grid points as well as at the grid points. These are different mathematical tasks.
A finite grid can be extended to an interval if extra regularity is established for both the target and the estimator or another interpolation rule; the box estimator itself jumps when a window crosses an observation. Merely drawing straight lines between grid endpoints does not prove a continuum band. A random grid or data-derived smoothness bound likewise needs an independent-data argument or a separate simultaneous theorem. The bound may be so wide as to be uninformative, especially for small or narrow ; nominal coverage is not the same as useful precision.
Common mistake
(Taking the smoothness bound from the same data)
The band needs to be fixed before the data are seen. Replacing it by the largest slope of does not work: a box estimate is piecewise constant, so its slope is zero between jumps and says nothing about for the true density. Any bound computed from the sample is random and dependent on the same observations, so the deterministic bias step of the proof no longer applies. Take from outside knowledge or from an independent sample, and state that assumption.
Common mistake
(Using the single-bandwidth radius after choosing among several)
If the radius is computed as if but the bandwidth is then chosen among five candidates, each pair fails with probability at most , and the union bound over the five candidates guarantees only probability for all of them. To keep , use with the true list size , and fix the list before looking at the data.
Exercises
Quick check
(For , and a window count of , compute the estimate and its exact variance in terms of the unknown window probability .)
Use the binomial count and keep symbolic.
Solution · Answer
The estimate is . The exact variance is . Replacing by produces a plug-in estimate, not the exact unknown variance.
Quick check
(With and , bound the smoothing bias. If the density is linear throughout the whole window, is that upper bound necessarily attained?)
Average the deviation over the symmetric window.
Solution · Answer
The general bound is . If throughout the window, the odd term averages to zero, so the actual bias is zero. The bound controls worst-case Lipschitz shapes and need not be attained for a particular density.
Quick check
(A fixed list has locations and bandwidths. What logarithmic factor is used at ? May the bandwidth be selected from the list after seeing the data?)
Count every location-bandwidth pair in the union bound.
Solution · Answer
The factor is . Yes: the simultaneous event already covers all five candidates at all ten fixed locations, so choosing among them using the same data preserves the stated lower coverage bound. This says nothing about a bandwidth outside the list.
Quick check
(For fixed and a fixed valid , give conditions on a sequence that make the displayed band radius converge to zero. Give one example.)
Control the two terms separately.
Solution · Answer
It suffices that and : then both the bias and displayed sampling radius vanish. For example works. These are sufficient conditions for this conservative bound; the variance calculation suggests that sharper arguments can work under weaker rate conditions.