From a label to a probability model
A classifier receives features and must choose a label. Naive Bayes models how those features are distributed within each class, then reverses the direction of conditioning to compare class probabilities. It is a model-based rule: its answer depends on both the estimated probabilities and the assumptions connecting features.
Consider two classes and two binary features . The model assumes and are conditionally independent given the class: for every class and values ,
This is not ordinary unconditional independence, and it must be justified or treated as an approximation. Each class must have positive prior probability for the conditional terms to be meaningful.
Scores and smoothing
Definition 3.1.1
(Unnormalized class score)
For observed and a class with , the unnormalized class score is
Proposition 3.1.2
(Posterior from class scores)
Let be the only two classes, with , and assume and are conditionally independent given the class. If , then for
Proof
(Total probability, then Bayes' rule)
By conditional independence, for each class, so is the joint probability . The classes partition the sample space, so the law of total probability gives , which is positive by assumption. Bayes' rule then states .
Since both classes share that positive denominator, the larger score gives the larger posterior. If the scores tie, a classifier needs an explicit tie rule. The normalized number is a posterior under the model; it is not automatically calibrated for new data.
Proof X-ray
Where the model and the partition enter
Two ingredients are used, and they play different roles. The partition into and supplies the denominator: it is what makes the probability of the observed features, so the same sum divides both scores. Conditional independence supplies the numerators: it is the only place where the product of two marginals is allowed to stand for a joint probability. The positive priors are needed so that conditioning on a class is meaningful, and the positive sum is needed to divide.
Notice what the argument does not establish. It shows that is the posterior of the model, and a ratio of scores is always meaningful as a comparison; but the numbers equal real joint probabilities only if the independence assumption holds for the data. A single score is also not a probability, since the two scores need not sum to one.
The next example shows what happens when the factorization is false.
Counterexample mode
Conditional independence is an assumption
Within a class, suppose but always. Then , while the product of marginals is . Multiplying marginals would misrepresent that joint distribution. Smoothing the marginals cannot remove this dependence.
Definition 3.1.3
(Binary Laplace smoothing)
Suppose the training sample has examples in class and with . Binary Laplace smoothing with parameter one estimates
The two numerators sum to , so these estimates form a distribution. The same rule is applied separately to and each class.
We use unsmoothed empirical class priors below; smoothing priors is a different explicit modeling choice. Smoothing prevents a feature value absent from a class sample from receiving estimated probability zero; it does not prove conditional independence.
Worked examples
Worked example 3.1.4
(A balanced two-class score)
There are eight training examples, four in each class. In class , three have and three have . In class , one has and two have . With empirical priors , the smoothed probabilities for the observed features are in and in . Thus
Their sum is , so the model posterior is . The score for is larger, so the model predicts under equal misclassification costs. The individual feature counts suffice for this model's calculation; they do not establish whether the independence assumption fits the joint training data.
Worked example 3.1.5
(A feature value never seen in one class)
Let each class have four training examples. In class , none has and two have ; in class , three have and one has . Priors are each and the observation is .
Without smoothing, and , so is ruled out completely. With smoothing, class has and , and class has and . Then and . The sum is , so and . The prediction is either way, but smoothing replaces an absolute exclusion by a posterior of for .
Worked example 3.1.6
(Changing the priors changes the decision)
Keep the smoothed likelihoods of the first example, for and for , but suppose is rare, with and . Then and . The sum is , so and . The prediction flips from to : the same features and the same likelihoods, but a different class balance.
Limits
Unequal classification costs can change the optimal decision rule even when posterior probabilities are fixed. Class imbalance changes the priors and hence the scores. A zero denominator makes the stated posterior undefined; with positive priors and parameter-one smoothing for each finite binary feature, both feature probabilities are positive, so this particular model avoids that issue. Evaluating generalization and calibration requires separate held-out data and is not established by a training calculation.
Common mistake
(Reading a score as a probability)
Reporting as "a 22% chance of " is wrong: is a joint probability of the class and the observed features, and the two scores sum to , not . Divide by first; in the first worked example the posterior of is .
Common mistake
(Forgetting the +2 in the smoothing denominator)
Adding one to the numerator but dividing by gives for and for when and no example has . These are not even probabilities, since and the two sum to . With the denominator the estimates are and , which sum to one.
Exercises
Quick check
(In the first worked example, compute and the posterior of .)
Use the scores and .
Solution · Answer
The sum is and .
Quick check
(If and no class- example has , what is the smoothed estimate of ?)
Apply the parameter-one rule with two possible values.
Solution · Answer
Parameter-one binary smoothing gives .
Quick check
(Under the dependence counterexample, compute and compare it with the product of marginals.)
Use and .
Solution · Answer
Since , the joint probability is , whereas the product is .
Quick check
(If two class scores are equal and positive, what are the two normalized posterior probabilities? What decision detail remains unspecified?)
Divide each score by their sum.
Solution · Answer
Both normalized probabilities are . A tie-breaking rule, or a decision rule reflecting unequal costs, still has to be specified.