5. Probability Foundations and Counting Models — Historical Enrichment

Learning goals and preparation

We assume fractions, finite sums, elementary sets, and Chapter 3's integration for the final continuous-model connection. We develop elementary probability from events and assumptions rather than from unexplained counting formulas. You will distinguish disjointness from independence, condition on information correctly, and derive the binomial model and its moments.

Historical optional enrichment. This chapter is an optional historical extension.

All numerical examples are hypothetical sampling or manufacturing models. Probabilities describe their assumptions; they do not certify an actual inspection process or an empirical defect rate.

5.1 Sample spaces and events

A sample space $\Omega$ lists possible outcomes of a modeled experiment. An event is a subset. A probability assignment satisfies $P(E)\ge0$, $P(\Omega)=1$, and additivity for disjoint events; in a general probability model this extends to countable disjoint unions. For a finite sample space it is enough to assign nonnegative probabilities to outcomes summing to one and add them over events.

Consequently $P(E^c)=1-P(E)$ and

$$P(E\cup F)=P(E)+P(F)-P(E\cap F).$$

To derive the second identity, split $E\cup F$ into disjoint pieces $E\setminus F$, $F\setminus E$, and $E\cap F$, then count the overlap only once.

Uniform counting, $P(E)=|E|/|\Omega|$, is valid only if the finite elementary outcomes are equally likely. A short list of named categories does not make them equally likely.

Worked example 1: categories are not outcomes. Two independent fair coin tosses have four equally likely ordered outcomes: HH, HT, TH, TT. The counts of heads are $0,1,2$, but these three categories are not equally likely. Their probabilities are $1/4,1/2,1/4$ because the middle category contains two ordered outcomes.

5.2 Counting with and without order

The multiplication principle counts sequential choices: if stage one has $m$ options and each has $n$ stage-two options, there are $mn$ ordered choices. Selecting $k$ distinct objects in order from $n$ gives $n(n-1)\cdots(n-k+1)=n!/(n-k)!$. Ignoring order divides by $k!$, because each selected $k$-element subset has exactly $k!$ orderings. Hence

$$\binom nk=\frac{n!}{k!(n-k)!},\qquad0\le k\le n.$$

Worked example 2: sampling without replacement. Among five objects, two are defective and three good. Select two uniformly without replacement. There are $\binom52=10$ equally likely unordered samples. The probabilities of two defective, one of each, and two good are respectively

$$\frac{\binom22\binom30}{10}=\frac1{10},\qquad \frac{\binom21\binom31}{10}=\frac6{10},\qquad \frac{\binom20\binom32}{10}=\frac3{10}.$$

They sum to one. Treating successive draws as independent would be wrong because the first changes the remaining composition.

5.3 Conditional probability and independence

For $P(B)>0$, define $P(A\mid B)=P(A\cap B)/P(B)$. It renormalizes the possible outcomes after restricting to event $B$. It follows that $P(A\cap B)=P(A\mid B)P(B)$.

Events $A,B$ are independent when $P(A\cap B)=P(A)P(B)$. When $P(B)>0$, this is equivalent to $P(A\mid B)=P(A)$. Disjoint events with positive probabilities are not independent: their intersection is empty while the product of their probabilities is positive.

Worked example 3: a changed second draw. In Worked example 2, the first draw is defective with probability $2/5$. Given that it was defective, the second is defective with probability $1/4$. Therefore both are defective with probability $(2/5)(1/4)=1/10$, matching the unordered count. The unconditional second-draw defect probability is still $2/5$ by symmetry, but its conditional value after observing the first differs.

If $B_1,\ldots,B_m$ partition the sample space with positive probabilities, then

$$P(A)=\sum_iP(A\mid B_i)P(B_i).$$

This is the law of total probability, obtained by adding the disjoint pieces $A\cap B_i$. If $P(A)>0$, Bayes' formula is

$$P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{\sum_iP(A\mid B_i)P(B_i)}.$$

Worked example 4: reverse an inspection condition. Suppose $2\%$ of a hypothetical production batch is defective. An inspection flags $90\%$ of defective objects and $5\%$ of good ones. A flag has probability $0.9(0.02)+0.05(0.98)=0.067$. Given a flag, the defect probability is $0.018/0.067=18/67$, about $0.269$. It is not $0.9$: the latter conditions on a different event. The many good objects generate substantial false flags despite their smaller individual flag probability.

5.4 Random variables and expectation

A random variable $X$ assigns a real number to each outcome. In a finite model, its distribution is $p_X(x)=P(X=x)$ and

$$E[X]=\sum_xxp_X(x),\qquad \operatorname{Var}(X)=E[(X-E[X])^2]=E[X^2]-E[X]^2.$$

The variance identity follows by expanding the square and using that probabilities sum to one. Expectation is linear: $E[aX+bY]=aE[X]+bE[Y]$, without needing independence. This follows by summing over the joint outcomes and distributing finite sums.

For two random variables with finite second moments, independence is sufficient for variance additivity; more generally, their variances add exactly when their covariance is zero. Expanding the variance of a sum produces a cross term $2(E[XY]-E[X]E[Y])$, which vanishes for independent variables. Identical marginal distributions alone do not remove that term.

Worked example 5: an elementary distribution. If $X$ has values $0,1,2$ with probabilities $1/4,1/2,1/4$, then $E[X]=1$, $E[X^2]=3/2$, and variance $1/2$. The expected value is a probability-weighted average, not necessarily the most likely value or a value observed on every experiment.

5.5 Bernoulli and binomial models

A Bernoulli variable $I$ equals one with probability $p$ and zero with probability $1-p$. Since $I^2=I$, it has mean $p$ and variance $p-p^2=p(1-p)$.

For $n$ independent trials with the same success probability $p$, let $X=I_1+\cdots+I_n$. A particular ordered outcome with $k$ successes has probability $p^k(1-p)^{n-k}$, and there are $\binom nk$ ways to locate them. Thus

$$P(X=k)=\binom nkp^k(1-p)^{n-k},\quad k=0,\ldots,n.$$

This is the binomial distribution. The probabilities sum to $(p+(1-p))^n=1$ by the binomial theorem. Linearity gives $E[X]=np$, and independence removes cross-covariances, giving variance $np(1-p)$. Sampling without replacement generally does not meet its independence assumptions.

Worked example 6: at least one event. In four independent trials with success probability $0.1$, the probability of at least one success is $1-(0.9)^4=0.3439$. The complement avoids adding four separate positive-count terms. Mean count is $0.4$ and variance $0.36$; neither is the probability of a particular outcome.

5.6 Connecting discrete and continuous models

A cumulative distribution function is $F(x)=P(X\le x)$. For a discrete variable it has jumps of size $P(X=x)$. In a continuous-density model, $F(x)=\int_{-\infty}^xp(t)dt$, with $p\ge0$ and total integral one. Then single points have probability zero, although intervals may have positive probability.

Expectation becomes an integral $\int xp(x)dx$ when it converges appropriately, as developed in Chapter 3. A density value is not a point probability, and a continuous graph is not a density until normalized. Discrete and continuous models are alternative mathematical descriptions whose suitability depends on the application.

5.7 Graded exercises

Foundation

1. If $P(A)=0.4$, $P(B)=0.5$, and $P(A\cap B)=0.2$, find $P(A\cup B)$ and decide independence.
2. How many unordered pairs can be selected from six distinct objects? How many ordered pairs without replacement?
3. Explain why disjoint events each of probability $1/3$ cannot be independent.

Core

4. A box has three red and two blue objects. Draw two without replacement. Find the probability both are red by conditional probability and by combinations.
5. If $P(B)=0.2$, $P(A\mid B)=0.8$, and $P(A\mid B^c)=0.1$, calculate $P(A)$ and $P(B\mid A)$.
6. Find mean and variance when $X$ takes values $0,2$ with probabilities $3/4,1/4$.
7. In five independent trials with success probability $1/2$, find $P(X=2)$, $E[X]$, and variance.
8. A Bernoulli variable $I$ has success probability $p$, where $0\le p\le1$. Compute $\operatorname{Var}(2I)$ directly and compare it with $2\operatorname{Var}(I)$. Explain when they differ.

Synthesis

9. A finite variable has CDF $F(x)=0$ for $x<0$, $1/4$ for $0\le x<1$, $3/4$ for $1\le x<2$, and one for $x\ge2$. Recover its distribution and expectation.
10. Explain which assumption fails if someone uses a binomial distribution for the number of defective objects in two draws without replacement from a small box. Using Worked example 2, compare the actual probability of two defects with the independent-trial prediction.

5.8 Full solutions

1. The union probability is $0.4+0.5-0.2=0.7$. Since $0.4(0.5)=0.2$, the two events are independent under the given probabilities.
2. Unordered pairs number $\binom62=15$. Ordered pairs without replacement number $6\cdot5=30$.
3. Their intersection probability is zero, while the product of their individual probabilities is $1/9>0$. This violates independence.
4. Conditionally, $(3/5)(2/4)=3/10$. By counting, $\binom32/\binom52=3/10$. The matching methods verify each other.
5. Total probability is $0.8(0.2)+0.1(0.8)=0.24$. Bayes gives $P(B\mid A)=0.16/0.24=2/3$.
6. $E[X]=2(1/4)=1/2$ and $E[X^2]=4(1/4)=1$. Variance is $1-1/4=3/4$.
7. $P(X=2)=\binom52/2^5=10/32=5/16$. Mean is $5/2$ and variance $5(1/2)(1/2)=5/4$.
8. Since $E[2I]=2p$ and $E[(2I)^2]=4p$, variance is $4p-4p^2=4p(1-p)$, whereas twice its original variance is $2p(1-p)$. They differ for $0<p<1$ and are both zero when $p=0$ or $p=1$. Scaling a variable scales squared deviations by the square of the scale. Equivalently $2I=I+I$ adds perfectly dependent copies in the nondegenerate case; at either endpoint the copies are constant and independent.
9. CDF jumps give probabilities $1/4$ at zero, $1/2$ at one, and $1/4$ at two. Expectation is $0+1/2+1/2=1$.
10. Successive draws are not independent because the remaining composition changes. With two defects in five objects, actual probability is $(2/5)(1/4)=1/10$. A binomial model with unchanged $p=2/5$ would give $(2/5)^2=4/25$, a different and unjustified value.

Summary and onward map

A probability model needs outcomes and explicit assumptions. Counting works only for equally likely elementary outcomes. Conditional probabilities restrict and renormalize; independence is not disjointness. Bayes reverses conditioning using base probabilities. Expectation is linear without independence, whereas variance addition needs dependence information. Binomial formulas require fixed-count independent identical trials; continuous-density formulas use integrals instead.