Introduction
Earlier notes developed Bayes' rule, measured classifiers with confusion matrices and error rates, and studied nearest-neighbor classification. A nearest-neighbor classifier estimates the class of a new point by looking at nearby training points. It does not commit to any model of how features arise within each class.
Naive Bayes does the opposite. It specifies a probability model for the features within each class, then uses Bayes' rule to turn that model into a posterior probability for the class. The specification is strong: within each class, the features are assumed independent. In return we get a classifier that is cheap to fit, easy to interpret, and whose evidence combines additively on the log-odds scale.
This note treats the binary case. Every feature is present or absent, and the class is one of two labels. We derive the classifier and state precisely what its assumptions do and do not say. We then show how it behaves on small samples and where it fails.
The model
Let be the class label, with class prior and . Let be a vector of binary features. For each feature and class define
Definition 5.4.1
(Conditional independence given the class)
The features are mutually independent conditional on if, for each and each ,
For a Bernoulli variable we can write . The exponent selects when and when . Hence the Bernoulli naive Bayes likelihood is
Because every lies strictly between and , every pattern has positive probability under both classes. In particular . By Bayes' rule, the posterior is prior times likelihood, normalized:
Additive log posterior odds
Theorem 5.4.2
(Log posterior odds of Bernoulli naive Bayes)
Under the model above, for every ,
Proof
(Cancel the evidence, then factor the likelihood)
Both posteriors share the same denominator , which is positive. Dividing one by the other therefore cancels it:
Substitute the factorized likelihood. The ratio of two products is the product of the ratios:
Every factor is a ratio of positive numbers, so the logarithm is defined. The logarithm turns products into sums and exponents into multipliers, which gives the stated formula.
Proof X-ray
Two hypotheses, used in two places
Positivity is used first. Because and , the evidence is positive, both posteriors are defined, and the common denominator can cancel; the same bounds make every factor positive, so every logarithm is finite. Conditional independence is used exactly once, to replace the joint likelihood ratio by a product of per-feature ratios. That is the only step that is a modeling choice rather than Bayes' rule and algebra.
The argument closes because the logarithm of a finite product of positive numbers is the sum of their logarithms. If the factorization is false, the first display still holds and only the second fails; the redundant-feature counterexample below shows that this single failure can change the prediction. If an estimated equals or , some factor can vanish and the logarithm is no longer finite, which is the zero-count problem treated under estimation.
In words, the formula is an accounting identity. The prior contributes , the base-rate term. Each feature then contributes one log-likelihood ratio. A present feature contributes , and an absent feature contributes . An absent feature is evidence too. If a feature is common in class 1 and rare in class 0, its absence pushes toward class 0.
Collecting the terms shows that the log odds equal , with
So the decision boundary is a hyperplane in . Contrast this with nearest neighbors, whose boundary can bend in whatever way the training points dictate.
What the assumption does and does not claim
Conditional independence is a statement within each class. It does not say that the features are independent marginally, once the classes are mixed together. The last exercise below shows that features independent within each class are usually dependent overall. The class acts as a common influence on all of them.
The assumption is also not a causal claim. It does not say that the features fail to influence one another in the world, or that the class "causes" them. It is a factorization of a joint distribution and nothing more.
If the model is the true distribution, then the posterior it computes is the true conditional probability . That conclusion is conditional on the model. A fitted model with estimated parameters, or a model whose independence assumption is false, produces numbers in that need not match real-world frequencies. Calibration must be checked, not presumed.
Decisions, costs and ties
A posterior is not yet a decision. Write . Let be the cost of predicting 1 when , and let be the cost of predicting 0 when ; correct decisions cost nothing.
Proposition 5.4.3
(Cost-sensitive decision threshold)
Assume and . Predicting 1 has strictly smaller expected cost than predicting 0 exactly when
Proof
(Compare the two expected costs)
Predicting 1 costs only when , so its expected cost is . Predicting 0 has expected cost . The inequality rearranges to , and dividing by the positive sum gives the threshold.
The threshold appears when the two costs are equal and positive. If is the correct posterior, the cost-sensitive rule minimizes expected cost; with equal positive costs it also minimizes misclassification error. An estimated or misspecified score has no such general guarantee. When missing a positive is costlier, the threshold drops below .
At equality the two actions have the same expected cost. The rule must still say what to do there. Throughout these notes we use a deterministic convention: a tie predicts class 0. With equal costs, that means posterior exactly , or log odds exactly , gives prediction 0.
A worked example
Worked example 5.4.4
(Two binary features with an imbalanced prior)
Take , , , and . The prior odds are . The two possible ratios for feature 1 are:
- present: ;
- absent: .
For feature 2 they are:
- present: ;
- absent: .
Pattern . The posterior odds are
The posterior is , and the prediction is class 1.
Pattern . The absent second feature contributes . The posterior odds are
The posterior is , and the prediction is class 0.
The same numbers can be organized as the linear score derived above, which also shows how the cost threshold moves the decision.
Worked example 5.4.5
(The linear score and a cost-sensitive cutoff)
With the parameters of the previous example,
The four patterns have posterior odds :
- : , posterior ;
- : , posterior ;
- : , posterior ;
- : , posterior .
The last two agree with the direct computation above. With equal costs only is classified as 1. If a missed positive costs three times a false alarm, , the threshold is ; now , with posterior , is also classified as 1, while , with , is not. An edge case: a third feature with has and contributes to , so it never changes the score, whether present or absent.
Common mistake
(Multiplying only over the present features)
For in the first example, multiplying only the present-feature ratio gives odds , a tie, instead of the correct . This is not Bernoulli naive Bayes with a step omitted. It is a heuristic score that discards the evidence carried by an absent feature, and it is not automatically a normalized probability model. Here it changes the reported score from to .
Numerical practice: logarithms and zeros
With hundreds of features, the product can fall below the smallest representable floating-point number, a failure called underflow. Summing logarithms avoids this. In practice, compute both log joint scores and , then compare them or take their difference.
The theorem required . Estimated parameters can violate that requirement. If an estimate equals and the corresponding feature is present, the computation asks for . Software may silently return or NaN, and the classifier then declares one class impossible on the strength of one feature. That outcome must be decided deliberately, not left to the arithmetic. The usual remedy is smoothing, developed below.
When conditional independence fails
Counterexample mode
Redundant features double-count evidence
Let . Within each class, let exactly, with and . Each marginal is then correct: and .
Truth for . Under the true model, and , since observing forces . The true posterior odds are . The posterior is , which under our tie rule predicts class 0.
Naive computation. Multiplying the marginals gives odds . The resulting posterior is , and the prediction is class 1.
The second feature repeats the first, but the naive model counts it as new evidence. Both the prediction and the stated confidence change, even though every individual parameter is estimated perfectly. Duplicated or highly correlated features (synonyms in text, derived variables) can double-count evidence and distort posterior probabilities; the direction of the distortion need not always be the same.
Impossible patterns. The pattern has probability under both classes in the true model. Then , and is undefined, because Bayes' rule would divide by zero. One should not condition on a pattern the model says cannot occur. The naive model assigns positive probability, which is another sign that it misdescribes this population.
Estimating the parameters
Suppose the training pairs are sampled independently from the population the classifier will serve. Let be the number of training cases in class , and let be the number of those cases with . Assume .
Maximum likelihood. For estimation, temporarily allow the closed parameter interval . The class- likelihood for feature is , which is maximized at . If estimation is restricted to the open interval used for the log-odds derivation, an all-zero or all-one sample has only a boundary supremum, not an attained maximum. When or , the estimate is or , and we are back at . A feature never seen in ten class-1 documents is not thereby impossible in class 1. A finite-sample zero count overstates certainty.
Theorem 5.4.6
(Beta posterior for a Bernoulli parameter)
Place a prior on , with . Given successes in independent class- trials, the posterior is , and its mean is
Proof
(Reading off the kernel)
The prior density is proportional to on . Multiplying by the likelihood gives
This is the kernel of . A density is determined by its kernel, so that is the posterior. A distribution has mean , which gives the formula.
With (a uniform prior), we get Laplace smoothing, . This always lies strictly inside .
Worked example 5.4.7
(A zero count with and without smoothing)
Suppose , and feature was never seen in class 1 but was seen times in class 0. Maximum likelihood gives and . For a new case with , the factor for this feature is , the log odds are , and the posterior is whatever the other features say.
Laplace smoothing gives and . The present-feature factor becomes and the absent-feature factor becomes . The feature now counts as strong but finite evidence for class 0, and other features can still outweigh it. The case needed no repair: under maximum likelihood its factor is , already finite.
A coherent predictive likelihood, under stated assumptions. Assume the following:
- independent Beta priors on all ;
- a known, fixed ;
- training observations independent given the parameters, with features conditionally independent given class and parameters; the new observation follows the same model and is conditionally independent of the training observations given its class and the parameters.
Then the likelihood and the prior both factor across , so the posteriors are independent across . Integrating against each posterior gives . So the product of these posterior-mean factors is exactly the posterior predictive . This justifies plugging in smoothed estimates as a Bayesian computation under these assumptions. It does not guarantee that the resulting probabilities are calibrated in reality.
The class prior is a separate quantity. The Beta hyperparameters belong to the feature parameters. The class prior needs its own treatment. It may be fixed from knowledge of the deployment population, or estimated, for example by under simple random sampling, or with its own Beta prior.
Common mistake
(Using the training class mix as the base rate)
If the training set was assembled with a deliberately balanced class mix, estimates that mix, not the deployment base rate. Plugging it in as shifts the base-rate term and every posterior with it, even when every is estimated well.
Evaluation discipline
Fit every data-dependent choice using training folds only: feature selection, binarization thresholds, imputation rules and the smoothing strength . Keep a final evaluation set untouched until the end. Assess classification metrics and probability calibration separately. A model can rank cases well while reporting badly inflated probabilities, as the redundant-feature example shows. Finally, the independence assumptions above do not cover grouped or repeated observations (several records from one person), temporal dependence, or a shift between training and deployment populations.
Exercises
Quick check
(Keep the feature parameters of the first worked example (, , , ) but take a balanced prior . Compute and the prediction under equal costs.)
Only the base-rate term changes.
Solution · Guided solution
The prior odds are now . The present feature 1 contributes and the absent feature 2 contributes , so the posterior odds are and
The prediction is class 1. With the same pattern had posterior and was classified as 0: the base-rate term changed from to while the feature evidence stayed the same.
Quick check
(Let , and suppose the total likelihood ratio for an observed is . Find the posterior and the equal-cost prediction.)
Convert the prior to odds before multiplying.
Solution · Guided solution
The prior odds are , so the posterior odds are . The posterior is
and the prediction is class 0. Evidence four times likelier under class 1 still leaves class 1 less probable, because the base rate starts at nine-to-one against. A large likelihood ratio does not by itself overcome a small prior. This is also why a classifier with good per-class detection rates can have low precision on rare classes.
Quick check
(Use one feature, fixed , , , and . Compute with Laplace smoothing and with maximum-likelihood plug-in estimates. Then explain what happens if a class has no training cases.)
Laplace smoothing adds one success and one failure to each class count.
Solution · Guided solution
The prior odds are .
Laplace smoothing. The estimates are and . The posterior odds are
so the posterior is .
Maximum likelihood. The estimates are and . The posterior odds are
so the posterior is .
Both predict class 1. Smoothing pulls each estimate toward and moderates the confidence. Neither number is automatically the real-world probability: both depend on assumptions, and the samples are tiny.
An empty class. If , then is undefined. The feature likelihood is constant in the parameter, so every allowed value maximizes it: there is no unique, data-identified maximum-likelihood estimate. Under a prior the posterior simply equals the prior, with mean (that is, under Laplace). That value is a statement of prior belief alone. It is not learned from data, and it should be reported as such.
Quick check
(Let , and , with conditionally independent given . Are and marginally independent?)
Mix the two classes before comparing with the product of marginals.
Solution · Guided solution
Each marginal is . By conditional independence within each class,
Marginal independence would require , so the features are marginally dependent. Equivalently, . The dependence comes from mixing the two classes. Seeing makes class 1 more likely, which in turn makes more likely.
Here the naive Bayes model is correct, so its posteriors are exact. Marginal dependence is fully compatible with the assumption. The redundant-feature counterexample is different in kind: there the dependence persists within each class, the factorization is false, and multiplying marginals double-counts the evidence.