Motivation
A posterior probability describes uncertainty about a label. It does not decide which mistake matters most. In a three-class problem, choosing the most probable class may expose us to a costly error that a less probable action avoids. A reject option can be sensible when every classification action has substantial conditional loss. These decisions should follow an explicit loss matrix rather than an informal confidence threshold.
The probabilities themselves usually have to be estimated. A parametric model shares information through a finite family, while a kernel density estimator averages nearby observations without imposing a fixed finite-dimensional density shape. Flexibility has a price: narrow neighborhoods reduce smoothing bias but use fewer effective observations. A density integrating to one is only the first requirement; it can still be a poor estimate.
This unit connects these two layers. We first prove the general Bayes decision rule and a stability bound for estimated posteriors. We then quantify the kernel estimator's bias, variance and mean integrated squared error on the real line. All sampling and smoothness assumptions are explicit, so the familiar bandwidth power is a conclusion of a calculation, not a universal prescription.
Definitions
Definition 20.1.1
(Actions, labels and conditional risk)
Let labels be and actions belong to a finite set , which may also contain rejection. Let and let be a fixed loss. Conditional risk and total risk are
A Bayes action minimizes this conditional risk, with a fixed measurable tie convention. Under zero-one loss it maximizes the posterior. If class priors are and class densities are , then wherever the denominator is positive.
Actions and labels need not be the same set. Rejection can have a constant loss for every true class; its conditional risk is then exactly . The posterior denominator is shared across ordinary actions, but costs still require summing across all true labels. Comparing a single weighted density for each action works only for particular losses, including zero-one loss.
Concept lensGeometric
Decision regions as cells of the probability simplex
Fix and let . The conditional risk is the linear function on the simplex of probability vectors. Action is Bayes exactly on the convex polyhedral cell , and the minimal risk is the lower envelope of these functions. Rejection at constant cost owns , also convex. Under zero-one loss this is the region where every .
Definition 20.1.2
(Parametric and kernel plug-in estimates)
A parametric plug-in rule estimates parameters in specified families and substitutes their densities and estimated priors into Bayes' formula. For independent observations from a density on , the kernel estimator is
Here is nonnegative, even, supported in a fixed bounded interval, integrates to one and has . Write and . Mean integrated squared error is .
In classification, each class uses its own sample size, not the total number of observations indiscriminately. Estimated priors and density estimates must form a nonnegative posterior with a positive denominator where a decision is made. Sparse local data can leave every estimated density zero under a compact kernel; one must declare how such observations are handled rather than divide by zero. The theorem below analyzes one density, so it does not by itself prove a classification convergence rate for a ratio of several estimated densities.
The benefit of a correct parametric assumption can be seen without an asymptotic theorem. For independent normal observations with known variance, their sample mean has mean squared error equal to that variance divided by sample size. Estimating the unknown center therefore has an exact inverse-sample-size accuracy. The density family has restricted the problem to one parameter; it is not trying to learn every smooth shape. This is useful information when the restriction is credible, rather than a reason to reject parametric methods categorically.
The same restriction can become persistent error under misspecification. Suppose the true distribution is an equally weighted mixture of unit-variance normals centered at opposite nonzero locations, but the working family permits only one unit-variance normal with an unknown mean. The fitted mean is still the sample mean. Its expectation is zero and its variance tends to zero, so Chebyshev's inequality makes it converge in probability to zero. The resulting working density approaches a centered unit-variance normal, while the true mixture has larger variance. Additional observations remove sampling fluctuation in the fitted center but cannot supply the missing shape parameter. A plug-in decision based on that family can consequently remain wrong even after its parameter estimates stabilize.
Kernel estimation trades this fixed shape restriction for a bandwidth-dependent averaging neighborhood. The variance calculation will show why the relevant effective sample size is proportional to the number of observations times the bandwidth. A larger sample permits both narrower smoothing and more observations in each shrinking neighborhood; taking the bandwidth to zero too quickly loses that benefit.
Theorem/Proposition
Theorem 20.1.3
(Bayes optimality and a plug-in risk bound)
A conditional minimizer minimizes total risk over measurable rules. If is a probability vector and minimizes its estimated conditional risk, then at every point
This bound also holds after expectation over an independent evaluation observation and any training randomness. It is a sufficient continuity bound, not a sharp characterization of every classification problem.
Theorem 20.1.4
(Pointwise kernel bias and variance)
Suppose is a bounded twice continuously differentiable density. As , at every fixed ,
and, for independent observations,
Consequently the variance equals as the bandwidth decreases. When and , both pointwise bias and variance tend to zero.
Theorem 20.1.5
(Integrated error and the one-dimensional bandwidth rate)
In addition suppose and . For and ,
The leading expression is minimized at , with leading error of order . The unknown roughness means this formula is not a directly available data-only bandwidth rule.
Proof sketch or proof idea
Proof
(Condition first, then compare estimated actions)
For every , the definition gives . Integrating proves optimality. Randomizing among actions gives a convex combination of their conditional risks and cannot improve the minimum. For a plug-in rule, add and subtract the estimated risks at and . The estimated-risk difference is nonpositive. Each remaining error has absolute value at most , proving the bound. Conditioning on a training set first and then taking its expectation gives the random-estimator statement.
Proof X-ray
Add and subtract the estimated risk
The real idea is to compare and through the estimated risk , which they are ordered by: because minimizes it. Writing as three differences leaves this nonpositive middle term and two outer terms, each of the form because the losses are bounded by . Boundedness is used only there. The factor counts the two outer terms. Nothing assumes is accurate; the bound just degrades with its error.
Proof
(Pointwise expansion and independence)
Changing variables gives . Taylor expansion on the bounded kernel support gives uniformly in . Evenness cancels the first-order term. The second moment of one summand, including its factor, is . Independence divides its variance by , proving the exact expression. Continuity and bounded support give the variance expansion. Averaging dependent observations would require covariance terms and would change this proof.
Proof
(Why the pointwise remainder also controls integrated error)
Pointwise Taylor expansion alone is insufficient for integrating an error over the whole real line. Use the exact integral remainder instead:
Translation is continuous in : approximate an function by a continuous compactly supported function, use uniform continuity for that approximation, and bound the two translated approximation errors by their unchanged norms. Minkowski's integral inequality bounds the distance from the displayed expression to by the double integral of . The integrand tends to zero and is bounded by , which is integrable over the kernel support and . Dominated convergence therefore proves convergence in . Squaring the norm gives the integrated squared bias expansion.
Integrating the exact variance and using nonnegativity to interchange integrals gives
Jensen's inequality under the probability weight yields , so the last term is and is negligible relative to . The bias–variance decomposition now proves MISE. Differentiating its leading two terms gives . This proves a rate for the stated smooth one-dimensional setting, not a universal lower bound or a multivariate formula.
Proof X-ray
A local Taylor theorem is not yet an integrated theorem
The pointwise bias uses continuity near one fixed location. MISE requires control of the entire line, supplied by square-integrable curvature and continuity of translations. The sampling variance additionally uses independence. Dropping any of these ingredients requires a replacement argument; the bandwidth exponent cannot be justified merely from the estimator's normalization.
Worked examples
Worked example 20.1.6
(Three classes and a cheaper non-MAP action)
Suppose posterior probabilities are and loss rows for actions one, two and three are , and . Conditional risks are two, , and . Action two is optimal although class one is most probable. A reject action costing for every label beats all three actions. Its use follows from the cost model, not a generic posterior confidence rule.
Worked example 20.1.7
(An explicitly computed kernel density)
Take on and zero elsewhere, data , and . At zero all three windows contribute, giving . At only observations zero and one contribute, giving . At three the estimate is zero. Its integral is one because each window contributes mass . These finite-sample steps are compatible with a density estimator; they do not make the underlying density discontinuous.
Worked example 20.1.8
(A normal density with a uniform kernel)
For this kernel, and . A standard normal density satisfies and , obtained by integrating . The leading optimal bandwidth is therefore . At zero the leading bias is , which is negative: smoothing lowers the peak. The leading variance is . Both contributions matter; minimizing only variance would push toward excessive smoothing.
Common mistakes
Counterexample mode
A support boundary breaks the smooth whole-line hypothesis
Let be uniform on and use the uniform kernel. At zero and , , while the chosen density version has . More substantially, on a boundary interval of width proportional to , smoothing loses a fixed fraction of the mass and creates integrated squared bias of order , not the stated . This density is not twice continuously differentiable on the whole line. Boundary correction is a different estimator with its own analysis.
For , this failure can be quantified exactly. On the expected estimator is , so integrated squared bias there is . On the true density vanishes and the smoothed spillover contributes another . The right boundary contributes the same two amounts, giving total integrated squared bias . Changing the density's value at a single endpoint cannot remove this interval-wide effect.
Common mistake
(Density accuracy and decision accuracy have different requirements)
A misspecified parametric family can converge to the wrong density despite stable parameter estimates. A kernel can estimate a smooth density well while a small posterior denominator makes ratios unstable. Conversely, sizable density errors far from any decision boundary may barely change labels. State the target, evaluation distribution and loss before interpreting an estimation error as a classification guarantee.
A bandwidth rate also does not certify the picture produced from one finite sample. The formula contains the unknown curvature, so in practice the bandwidth must be estimated or chosen by validation, and a rare class cannot borrow the total sample size to pretend that it has many observations of its own. To compare two classifiers, compute the specified cost under the same evaluation distribution rather than comparing the heights of density peaks. That an estimate integrates to one guarantees only that it is a density; the bias and variance calculation answers how accurate it is; and the loss incurred after the density is substituted into a decision is a further layer to examine. The three checks are connected, but none replaces the other two.
Tuning the bandwidth and reporting the best cost on the same data also introduces selection optimism; the tuning procedure should be placed inside an independent evaluation.
Summary
Bayes decisions minimize posterior-weighted cost across all labels. Plug-in error propagates through conditional risk, but density estimation and decision making remain separate tasks. Independent one-dimensional kernel samples yield a squared-bias term of order and a variance term of order under explicit smoothness assumptions. Balancing them gives the stated bandwidth and error rates, with boundary and sampling limitations that matter.
Exercises
Quick check
(For posterior , retain the three loss rows from the example. For which nonnegative constant rejection costs is rejection a Bayes action?)
Include the tie case and distinguish it from unique optimality.
Solution · Answer
The smallest ordinary risk is . Rejection is Bayes for , uniquely best for , and tied with action two at equality. For , action two is better. Posterior maximum alone would not determine this threshold under the stated asymmetric losses.
Quick check
(Compute the second moment and squared integral of the uniform kernel, and derive the leading bandwidth for a density with curvature roughness A greater than zero.)
Integrate on minus one to one.
Solution · Answer
and . If , the leading error is . Its derivative vanishes at , so . Differentiating again or noting the derivative changes sign proves the positive minimum.
Quick check
(If h equals n to the power minus one quarter, determine the orders of integrated squared bias and integrated variance. Which dominates? Compare with the optimal exponent.)
Use the proven expansion rather than a normalization argument.
Solution · Answer
The squared-bias order is and variance order is . Variance dominates, yielding the slower order under nonzero leading constants. The balanced choice proportional to gives both terms order . Both bandwidths vanish and have diverging effective sample size, but that does not make their accuracy rates equal.
Quick check
(Suppose costs are bounded by five and posterior L1 error is at most 0.01 everywhere. Bound excess conditional and total risk. Does this prove the labels always agree?)
A risk bound is not an exact decision-stability assertion.
Solution · Answer
The theorem bounds conditional excess by , and expectation gives the same total-risk bound. Labels can still differ near a conditional-risk tie, where a small posterior perturbation changes the minimizing action. To guarantee agreement requires a positive risk gap larger than an appropriate estimation-error bound, not merely a small global upper bound on excess risk.