Motivation
Probability provides a language for incomplete information. The same outcome can belong to several events, so counting outcomes without accounting for overlap is dangerous. The model also separates the information available before an observation from the information available afterwards.
Definitions
Events and the model
A probability calculation starts before arithmetic. One chooses a sample space , whose points describe outcomes at the resolution needed for the question. An event is a set of outcomes. For a finite experiment every subset can be an event; for an infinite space we instead specify a collection containing and closed under complements and countable unions. This restriction permits a consistent theory even when not every subset can be assigned a probability.
Definition 1.1.1
(Probability space)
A probability space is a triple , where is a collection of events as above and satisfies , , and, for pairwise disjoint events ,
The last property is countable additivity. The model is a mathematical assumption: equally likely outcomes, for example, must be justified by the experiment, not inferred from the word “random”.
Theorem 1.1.2
(Consequences of the axioms)
For events , and . If , then .
Proof
(Disjoint decompositions)
Countable additivity applies to finitely many disjoint events by taking from some index on; because would otherwise have infinite probability. First, with disjoint terms gives . Next decompose as the disjoint union of and , so . Since is itself the disjoint union of and , we have , and substitution gives the addition rule. Finally, if then with disjoint terms, so because probabilities are nonnegative. The subtraction in the addition rule corrects for counting the overlap twice.
Worked example 1.1.3
(A non-uniform sample space)
A device reports with respective probabilities . Let and . Then , , and . The addition rule gives . Counting two outcomes out of three for would give the wrong answer because the outcomes are not equally likely.
Proof sketch or proof idea
The argument above derives useful identities from the axioms rather than treating each formula as an independent rule. Its disjoint decompositions will recur when observations are introduced.
Conditioning and partitions
Definition 1.1.4
(Conditional probability)
When an event with is known to have occurred, the conditional probability of given is
It renormalizes the probability inside ; it does not change the original probability measure. In particular, the displayed ratio is undefined if . More advanced conditional distributions can handle some zero-probability observations, but the elementary ratio alone cannot do so.
Concept lensStructural
Conditioning produces a new probability measure
Fix with and write . Then is itself a probability measure on : it is nonnegative, , and for disjoint the sets are disjoint with union , so countable additivity passes to . Every consequence of the axioms therefore holds conditionally, for example . The new measure gives probability zero, and conditioning it once more on with gives
so two successive observations amount to one observation of their intersection.
Rearranging gives the multiplication rule . Applied cell by cell to a partition, it gives the two formulas used for updating a model after an observation.
Theorem 1.1.5
(Total probability and Bayes' formula)
Let partition with for every , and let be an event. Then
If moreover , then for each ,
Proof
(Splitting the observation along the partition)
The sets are disjoint, because the are, and their union is . Additivity and the multiplication rule give . When , the definition gives , and substituting the first formula for gives Bayes' formula.
The denominator accounts for every way to observe , not just the hypothesis being investigated. A zero-prior cell contributes zero and can be omitted from the conditional-probability version of the sum.
Worked example 1.1.6
(Updating after an imperfect signal)
A component comes from line with probability and from line otherwise. A warning occurs with probability on and on . The warning probability is . Given a warning, the probability of is . The warning doubles the prior probability of , but does not make it certain.
Concept lensAlgebraic
Bayes' formula as prior odds times a likelihood ratio
For two hypotheses with and , divide Bayes' formula for by the one for ; the common denominator cancels:
Posterior odds are prior odds multiplied by the likelihood ratio. In the example the prior odds are and the ratio is , so the posterior odds are , that is, .
Independence is a statement about the model
Definition 1.1.7
(Independence and mutual independence)
Events and are independent when . Events are mutually independent when
for every choice of indices with .
For two events, if , the product rule is equivalent to . The product definition also makes sense for a zero-probability event. Independence is symmetric, whereas the notation for conditioning has a direction. Events with positive probabilities that are mutually exclusive cannot be independent: their intersection has probability zero while the product is positive.
For several events, mutual independence requires the product rule for every subcollection, not merely every pair.
Counterexample mode
Pairwise independence is not mutual independence
Claim: if each pair among is independent, then the three events are mutually independent. Take two fair independent bits , so the four pairs each have probability , and let , , . Each event has probability . Every pairwise intersection is the single outcome : , and all equal , with probability . Yet also has probability , not . The claim fails; mutual independence of three events needs the three pairwise identities and the triple identity, which is exactly what the definition lists.
Knowing any two of these bits' events determines the third, which is why the triple product fails.
Repeated sampling and the denominator
The sample space must record the method of sampling. In particular, drawing without replacement changes the conditional distribution after the first draw. Drawing with replacement restores the original composition before each draw, but independence still belongs to the model rather than to the word “replacement” alone.
Theorem 1.1.8
(Chain rule for a sequence of events)
If , then
Proof
(Telescoping the multiplication rule)
Write . For we have , so monotonicity gives ; every conditional probability in the product is therefore defined. By the definition of conditional probability, , and
If a required conditioning event has probability zero, this particular formula is not defined, even though the intersection itself then has probability zero.
Worked example 1.1.9
(Two draws without replacement)
A bag contains three blue and two green balls. Draw twice uniformly without replacement. The probability of two blues is : after a blue is removed, two of the four remaining balls are blue. The probability of exactly one blue is , where the two terms are disjoint orders. Replacing the ball between draws would instead give for two blues. The distinction is a change of experiment, not a correction to arithmetic.
A useful check is that the conditional probabilities of all possible next outcomes sum to one. Here, after a blue, the next draw is blue with probability and green with probability . Conditional probabilities should always be computed within the stated conditioning event rather than mixing denominators from different stages.
Common mistake
(Reversing a conditional probability)
and generally have different denominators. In the warning example the former might describe the source given the warning; the latter describes the warning given the source. Bayes' formula converts between them only after the relevant prior and total probability are supplied.
Counting equally likely outcomes
When all outcomes in a finite sample space have the same probability, the axioms force each to have probability , and . The equality follows by adding the probabilities of the singleton outcomes in . The essential work is choosing a sample space whose outcomes really are equally likely. For example, when two fair dice are distinguishable by color, the 36 ordered pairs are equally likely. The sums from two to twelve are not equally likely, since different sums arise from different numbers of pairs.
Worked example 1.1.10
(A sum from two dice)
The sum seven occurs for , six out of 36 equally likely ordered pairs. Its probability is . By contrast sum two occurs only for and has probability . Treating the eleven possible sums as equally likely would give for each, contradicting the experiment's ordered-pair model.
Order is not always observable, but an ordered representation can still be useful. When every unordered selection of a given size corresponds to the same number of ordered sequences, either representation works if numerator and denominator use the same convention. Problems arise when ordered counts appear in the numerator and unordered counts in the denominator, or when unequal multiplicities are silently treated as equal.
Suppose three distinct balls are selected uniformly without replacement from ten, of which four are red. The probability of exactly two red is
The numerator first chooses the two red balls and then the one non-red ball. It counts each unordered selection once. Alternatively, the ordered calculation adds the three placements of the non-red draw; each has probability , producing the same result. A quick consistency check is that probabilities for zero, one, two, and three red balls sum to one.
Complements and repeated trials
Sometimes “at least one” is easier through its complement. If independent trials have the same event probability , the probability of at least one success in trials is . The complement is the event that all trials fail, whose probability is a product by mutual independence. The formula is not justified by equal marginal probabilities alone. For example, if the same fair coin outcome is reused in every trial, the probability of at least one head remains , not .
Worked example 1.1.11
(A detection opportunity)
An instrument independently detects a signal on each of four passes with probability . The chance of at least one detection is . The chance of exactly one detection is : choose which pass succeeds, and multiply the probability for that order. These two questions differ; “at least one” also includes two, three, or four detections.
Common mistake
(Adding the probabilities of overlapping successes)
It is tempting to say that four passes, each detecting with probability , detect at least once with probability . A number above one cannot be a probability. The events “pass detects” overlap, so their probabilities may be added only after the overlaps are subtracted. The complement route avoids this: . The sum is only an upper bound for the probability of at least one success.
Conditioning can change dependence
Independence can disappear when a third event is observed. Take two independent fair bits and condition on the event that at least one is one. The three outcomes are then equally likely. Each bit equals one with conditional probability , while both equal one with conditional probability . Since , the bits are not conditionally independent given this event. The observation excludes and thereby links the possible values. Independence before an observation does not automatically survive the restriction to an observed event.
The reverse phenomenon can also occur: dependence in a mixture may vanish after conditioning on the common cause. A model should state which probabilities are conditional and on what information. Bayes' formula does not require independence; it follows from the multiplication rule and a partition. Independence is an extra assumption that may simplify one or more terms, and should be justified rather than inserted to make the arithmetic convenient.
Checking a model with bounds
Before substituting into a formula, check whether its input probabilities could arise from any joint model. Since is contained in each event, . The addition rule and also give . Nonnegativity supplies a second lower bound of zero. Combining them yields
For instance, if and , an intersection probability of is impossible: the lower bound is . An intersection of is consistent with these bounds. Indeed one can assign probabilities to , to , to , and to neither event. The four cells are nonnegative and sum to one, so they define a finite model. This construction shows consistency for this particular choice, not independence: independence would require intersection .
If , division of the intersection bounds by also bounds . Such checks catch impossible inputs before a calculation is carried through. They do not identify the exact intersection from the two marginal probabilities alone; additional information or assumptions are needed.
Consider and once more. If the events are independent, their intersection is and their union has probability . If instead is a subset of , the intersection is and the union has probability . Both constructions have the same two marginal probabilities. Thus a question specifying only those marginals admits different answers for the union: the relationship between the events is part of the problem's data. Draw a four-cell table whenever it is unclear which quantities are known and which are being assumed.
Summary
Events are sets in a specified probability model. Conditioning renormalizes the part of the model consistent with an observation; independence states a product identity. Disjointness and independence are different, and pairwise independence is weaker than mutual independence.
Exercises
Quick check
(A fair die is rolled. Let be an even result and a result greater than three. Find and decide whether and are independent.)
Solution · Answer
and , so . Since , the events are not independent.
Quick check
(A box is selected with probabilities and . A red ball has conditional probability from the first box and from the second. Find the probability of the first box given red.)
Solution · Answer
By total probability, red has probability . Bayes' formula gives for the first box.
Quick check
(If , , and , compute and .)
Solution · Answer
Inclusion-exclusion gives . Partitioning into and gives .