A probabilistic classifier does more than attach a label to a feature vector: it models how strongly the observed features support each class. That distinction matters when errors have different costs or when two predictions have similar probabilities. It also makes the assumptions visible. Multiplying feature probabilities is legitimate only under a specified joint model.
We will study binary features, derive the naive Bayes posterior, and turn its product into a linear log-odds score. An exact count example will show why an absent feature still contributes evidence. Smoothing prevents zero-count probabilities, while explicit dependence examples show what smoothing cannot repair.
Start with conditional risk, not a classification slogan
Let be a class and a binary feature vector. At an observed vector , write .
Proposition 3.3.1
(Minimum conditional risk)
Let , so . Suppose a false positive costs , a false negative costs , and correct predictions cost zero. Predicting 1 has conditional risk , and predicting 0 has conditional risk . Hence the prediction of smaller risk is 1 exactly when , that is, when
With equal costs this is the maximum-posterior rule. Equal risks are resolved by the convention that chooses class 0.
Proof
(Compare the two conditional risks)
Predicting 1 is wrong only when , which has probability and cost ; predicting 0 is wrong only when , with probability and cost . Choose 1 exactly when . Substituting gives , and dividing by the positive number gives the stated threshold. If both costs are one, the risk of predicting is , so minimizing risk is maximizing the posterior.
Proof X-ray
What the threshold depends on
The only step that uses more than the definition of risk is the substitution , which needs the two classes to exhaust the outcomes. Positivity of both costs is used when dividing by . The value one half appears only when the two costs are equal: it comes from the cost assumption, not from probability notation alone. At equality the risks coincide, so the tie convention is a choice, not a consequence.
The proposition is relative to the probabilities supplied to it. If a fitted model's posterior differs from the true conditional probability, the rule is optimal for the model, not for the data.
These are optimal decisions for the probabilities supplied to the rule. If the fitted probability model differs from the data-generating distribution, its posterior need not be the true conditional probability.
Define the Bernoulli naive Bayes model explicitly
Definition 3.3.2
(Class priors and conditional independence)
Let , with , and let
Bernoulli naive Bayes assumes that the features are independent conditional on the class:
For use the factor ; for use . The formula describes binary presence/absence observations, not occurrence counts.
For each class, summing this product over all binary vectors gives , so it defines a normalized conditional model. Independence is an assumption about the joint distribution; fitting separate feature probabilities does not prove it.
Conditional independence is also different from independence without conditioning. If a fair binary class has , both features are deterministic and independent once the class is fixed. Without conditioning, however, , while the product of their marginal probabilities is .
Bayes' rule gives unnormalized class scores
provided . Comparing suffices for equal-cost classification because they share the same positive denominator.
Estimate probabilities from counts, then say what smoothing changes
Assume each class has labeled training examples, and let count the examples of class with feature equal to one. The empirical class-prior estimate is , where .
For one feature and class, the independent Bernoulli log likelihood is
When , setting its derivative to zero gives . The second derivative is , so this is the unique maximum. For or , the maximum occurs at 0 or 1 respectively.
Those boundary estimates can cause trouble: one unobserved feature outcome can set a whole class score to zero. If every class assigns zero probability to the new vector, posterior normalization is undefined.
Definition 3.3.3
(Symmetric pseudocount smoothing)
For , add to both the one-count and the zero-count:
The denominator contains because a binary feature has two outcomes. The probabilities sum to one and are strictly between zero and one. Unit pseudocounts, , are often called Laplace smoothing.
This defines a smoothed estimate, rather than the ordinary maximum-likelihood estimate from the observed examples alone. Together with positive class priors, it makes both class scores positive for every binary vector. It does not establish conditional independence or correct a misspecified model. As grows large with the counts fixed, each feature probability approaches one half, reducing its class-specific influence.
Worked example 3.3.4
(An exact posterior uses both present and absent features)
Use the following synthetic counts, with four examples in each class:
| Class | Feature-one count | Feature-two count | |
|---|---|---|---|
With , priors are both and feature probabilities are
For , the second feature is absent. The scores are
Therefore the fitted posterior of class 1 is . The equal-cost rule predicts class 1.
If we drop the absent-feature factor, the scores instead become and , producing posterior . That is a different calculation. In a Bernoulli model, absence is an observed outcome with its own probability. A multinomial count model uses a different likelihood; its rules cannot be substituted silently.
For posterior with false-positive cost 5 and false-negative cost 1, predicting 1 costs , while predicting 0 costs . The cost-sensitive rule therefore chooses 0. The most probable class and the least costly decision can differ.
The explorer below recomputes this worked model from its counts, so you can check the factor contributed by each observed absence, change the pseudocount, and see that changing error costs alters the decision but not the fitted posterior.
Worked example 3.3.5
(Smoothing repairs a zero-count posterior)
Use four examples in each class and two features. In class 0, feature one occurs times and feature two times; in class 1, feature one occurs times and feature two times. Both priors are , and the vector is .
The unsmoothed estimates are and . Class 0 gets because feature one is present, and class 1 gets because feature two is present, so and the posterior is undefined.
With , the probabilities of a one are , , and . Then
The posterior of class 1 is , so the equal-cost rule predicts class 0. Smoothing made the calculation well defined; it did not make the independence assumption true. This is the boundary case in which every unsmoothed score vanishes.
Common mistake
(Adding the pseudocount only to the numerator)
For , , , the estimates for a one and for a zero are not probabilities: they sum to . Each of the two outcomes receives , so the denominator must be , giving and .
Turn the product into a linear log-odds score
With positive priors and feature probabilities strictly inside , all relevant logarithms exist.
Proposition 3.3.6
(Naive Bayes log odds are linear in the features)
Suppose and for every and . Then for every binary vector
where
Proof
(Take the ratio of the two scores)
The posterior normalizer is common to both classes, so the ratio of posteriors is . Taking logarithms of the two products gives
In the second line, expand as . The terms multiplying combine into , and the remaining terms form .
Thus binary naive Bayes has a linear log-odds score, even though it was introduced through class-conditional products. Equal-cost classification chooses 1 when this score is positive. With unequal costs, compare it instead with . The worked vector has log odds ; cost ratio five gives a larger threshold .
To compute probabilities from two log scores , set and normalize . The largest exponential is one, so both scores cannot disappear through multiplying many small probabilities. The common factor cancels mathematically; the implementation computes floating-point approximations to the fitted probabilities.
Worked example 3.3.7
(The log-odds score reproduces every posterior, including ties)
For the counts of the first worked example with , the smoothed probabilities are and , with equal priors. The weights are
and the constant is . Therefore the log odds of class 1 are .
| Vector | Log odds | Posterior of class 1 | Equal-cost prediction |
|---|---|---|---|
| (tie) | |||
| (tie) |
Odds give posterior , odds give , and odds give . Direct scores agree: at and both and equal . The two tie vectors show that a zero score is an exact tie, which the stated convention resolves to class 0. The two features here have opposite weights because feature one favors class 1 and feature two favors class 0.
Concept lensGeometric
A hyperplane through the binary cube
Geometrically, the fitted rule is a half-space test. For a threshold ( for equal costs, otherwise), the set is a hyperplane in with normal vector , and class 1 is predicted at the corners of strictly on its positive side. Changing the costs slides the hyperplane without turning it. In the worked model and , so the boundary for equal costs is , and the tie vectors and lie on it. For cost ratio five the boundary is , beyond the corner , so no corner is assigned to class 1.
Common mistake
(Zero log odds is the threshold only for equal costs)
For the vector above the log odds are , which is positive, so equal costs predict class 1. With and the threshold is , so the same vector is assigned to class 0. The score is a property of the fitted model; the threshold belongs to the decision problem.
Dependence can change confidence or destroy the useful signal
Counterexample mode
Duplicating a feature can count the same evidence twice
Suppose the true distribution has equal class priors and exactly. Let and .
At vector , the true class-conditional probabilities are and : the second feature adds no new information. The true posterior of class 1 is .
The naive factorization instead squares those probabilities, giving posterior
The predicted class is unchanged, but confidence is inflated. Smoothing zero counts does not remove this duplicated evidence.
For a different failure, let be independent fair bits and let . In each class, each feature separately has probability one half of being one. A naive model using those exact marginals gives posterior one half for every vector, although the pair determines the class perfectly. Here unconditional feature independence holds, but conditional independence fails. The interaction, not either feature's individual frequency, carries the label information.
An implementation that stores log probabilities directly
The code below uses unit pseudocounts and empirical class priors. It requires both classes in the training data and equally sized nonempty binary rows. Log probabilities for zero outcomes are computed from their smoothed counts, rather than by subtracting a rounded probability from one.
from math import exp, log
def fit_bernoulli_nb(rows, labels):
rows = [tuple(row) for row in rows]
labels = tuple(labels)
if not rows or len(rows) != len(labels):
raise ValueError("one label is required for every training row")
dimension = len(rows[0])
if not dimension or any(len(row) != dimension for row in rows):
raise ValueError("rows must have the same positive dimension")
if (any(value not in (0, 1) for row in rows for value in row)
or any(label not in (0, 1) for label in labels)):
raise ValueError("features and labels must be binary")
counts = [labels.count(y) for y in (0, 1)]
if 0 in counts:
raise ValueError("both classes must occur in the training data")
log_prior, log_one, log_zero, one_counts = [], [], [], []
for y in (0, 1):
ones = [sum(int(row[j]) for row, label in zip(rows, labels)
if label == y) for j in range(dimension)]
one_counts.append(ones)
denominator = log(counts[y] + 2)
log_prior.append(log(counts[y]) - log(len(rows)))
log_one.append([log(s + 1) - denominator for s in ones])
log_zero.append([log(counts[y] - s + 1) - denominator for s in ones])
return log_prior, log_one, log_zero, counts, one_counts
def predict_bernoulli_nb(features, model):
log_prior, log_one, log_zero, counts, one_counts = model
features = tuple(features)
if (len(features) != len(log_one[0])
or any(value not in (0, 1) for value in features)):
raise ValueError("features must be a binary row of the trained dimension")
scores = [log_prior[y] + sum(
log_one[y][j] if value else log_zero[y][j]
for j, value in enumerate(features)) for y in (0, 1)]
maximum = max(scores)
weights = [exp(score - maximum) for score in scores]
total = sum(weights)
posterior = [weight / total for weight in weights]
# Compare exact rational scores after canceling the common factor 1 / N.
numerators = []
for y in (0, 1):
numerator = counts[y]
for j, value in enumerate(features):
ones = one_counts[y][j]
numerator *= ones + 1 if value else counts[y] - ones + 1
numerators.append(numerator)
dimension = len(features)
prediction = int(
numerators[1] * (counts[0] + 2) ** dimension
> numerators[0] * (counts[1] + 2) ** dimension
)
return prediction, posterior
The class decision uses exact integer arithmetic for this particular model: after canceling the common factor , the score for class is times the product of the selected smoothed counts, divided by . Cross multiplication compares these rational scores without a tolerance. Directly comparing floating-point log scores cannot reliably identify a mathematical tie. For example, one class-0 row (1,1) and four class-1 rows (0,1), (0,1), (0,1), (0,0) give at (1,1); rounded log scores can nevertheless differ. The returned posterior still uses floating-point log scores and is approximate, including at a tie.
One dataset realizing the worked counts has class-0 rows (0,1), (0,1), (1,1), (0,0) and class-1 rows (1,0), (1,0), (1,1), (0,0). For (1,0), the implementation returns class 1 and probabilities approximately [0.2, 0.8].
For rows and features, fitting uses basic counting operations and stores model entries; this teaching implementation also materializes the input rows. Prediction uses floating-point operations and integer multiplications, but its exact integer products grow with and , so their bit cost is not constant. The prediction function implements equal-cost classification; the approximate posterior supports a separate explicitly chosen cost-sensitive decision.
Exercises and worked solutions
Quick check
(1. Normalize smoothed binary counts)
For one feature with , and , compute the probabilities of one and zero. Why is the denominator six rather than five?
Solution · Solution 1
The probabilities are and . There is one pseudocount for each of two outcomes. Using denominator five for both would make their sum , so they would not be a probability distribution.
Quick check
(2. Keep absent-feature evidence)
In the worked count table, give the class scores and posterior for . What posterior results if the absent second feature is incorrectly omitted?
Solution · Solution 2
The correct scores are and , giving posterior for class 1. Omitting the zero-outcome factors gives scores and posterior . Both presences and absences are modeled Bernoulli outcomes.
Quick check
(3. Change costs without changing the posterior)
Given posterior , false-positive cost five and false-negative cost one, compute both decision risks and the optimal class. What is the threshold on ?
Solution · Solution 3
Predicting 1 has risk , while predicting 0 has risk , so choose 0. The threshold is , above the posterior . The probabilities stay the same; the decision objective changed.
Quick check
(4. Distinguish the two meanings of independence)
If a fair class has , are the features independent conditional on ? Are they independent without conditioning? Compare the relevant probabilities.
Solution · Solution 4
Given the class, both features are fixed, and their conditional joint probabilities factorize, so conditional independence holds. Marginally, the probability of (1,1) is , but the product of marginal one-probabilities is . Thus marginal independence fails.
Quick check
(5. Count duplicated evidence accurately)
In the duplicated-feature counterexample, compute the true and naive posteriors at (1,1). Why does a high naive posterior not prove the features supplied two independent pieces of evidence?
Solution · Solution 5
The true posterior is , while the naive posterior is . The second feature is an exact copy of the first, so it supplies no additional conditional information. Multiplying both factors assumes precisely the independence that fails in this example.
Quick check
(6. Explain a missing interaction)
For independent fair input bits labeled by XOR, what are the feature-one probabilities within each class? What posterior does their naive factorization produce, and why can the original pair still classify perfectly?
Solution · Solution 6
Every conditional feature-one probability is , and the class priors are also . Both class scores equal for every vector, so the posterior is [1/2, 1/2]. Yet checking whether the bits differ gives the label exactly. The lost information is their class-dependent joint pattern, which the conditional product model does not represent.