9. Probability, conditioning and finite random variables

Learning goals and finite models

Specify a finite sample space, compute conditional probabilities, distinguish disjointness from independence, and derive mean and variance for Bernoulli sums. A sample space $\Omega$ contains all modeled outcomes. Assign nonnegative weights $p(\omega)$ summing to one and define $P(A)=\sum_{\omega\in A}p(\omega)$. Only when all weights are equal does $P(A)=|A|/|\Omega|$.

A sample space $\Omega$ lists possible outcomes of a modeled experiment. An event is a subset. A probability assignment satisfies $P(E)\ge0$, $P(\Omega)=1$, and additivity for disjoint events; in a general probability model this extends to countable disjoint unions. For a finite sample space it is enough to assign nonnegative probabilities to outcomes summing to one and add them over events.

Consequently $P(E^c)=1-P(E)$ and

$$P(E\cup F)=P(E)+P(F)-P(E\cap F).$$

To derive the second identity, split $E\cup F$ into disjoint pieces $E\setminus F$, $F\setminus E$, and $E\cap F$, then count the overlap only once.

Uniform counting, $P(E)=|E|/|\Omega|$, is valid only if the finite elementary outcomes are equally likely. A short list of named categories does not make them equally likely.

The multiplication principle counts sequential choices: if stage one has $m$ options and each has $n$ stage-two options, there are $mn$ ordered choices. Selecting $k$ distinct objects in order from $n$ gives $n(n-1)\cdots(n-k+1)=n!/(n-k)!$. Ignoring order divides by $k!$, because each selected $k$-element subset has exactly $k!$ orderings. Hence

$$\binom nk=\frac{n!}{k!(n-k)!},\qquad0\le k\le n.$$

For $P(B)>0$, define $P(A\mid B)=P(A\cap B)/P(B)$. It renormalizes the possible outcomes after restricting to event $B$. It follows that $P(A\cap B)=P(A\mid B)P(B)$.

Events $A,B$ are independent when $P(A\cap B)=P(A)P(B)$. When $P(B)>0$, this is equivalent to $P(A\mid B)=P(A)$. Disjoint events with positive probabilities are not independent: their intersection is empty while the product of their probabilities is positive.

If $B_1,\ldots,B_m$ partition the sample space with positive probabilities, then

$$P(A)=\sum_iP(A\mid B_i)P(B_i).$$

This is the law of total probability, obtained by adding the disjoint pieces $A\cap B_i$. If $P(A)>0$, Bayes' formula is

$$P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{\sum_iP(A\mid B_i)P(B_i)}.$$

Random variables and expectation

A random variable $X$ assigns a real number to each outcome. In this finite model, its distribution is $p_X(x)=P(X=x)$ and

$$E[X]=\sum_xxp_X(x)=\sum_{\omega\in\Omega}X(\omega)p(\omega),\qquad \operatorname{Var}(X)=E[(X-E[X])^2].$$

The two expressions for the expectation agree by grouping outcomes with the same $X$ value. For real constants $a,b$, distributing the finite outcome sum gives

$$E[aX+bY]=\sum_{\omega}(aX(\omega)+bY(\omega))p(\omega)=aE[X]+bE[Y].$$

Thus linearity needs no independence. With $\mu=E[X]$ and $E[1]=1$, expanding the square gives

$$\operatorname{Var}(X)=E[X^2]-2\mu E[X]+\mu^2E[1]=E[X^2]-E[X]^2.$$

Define the joint distribution by $p_{X,Y}(x,y)=P(X=x,Y=y)$, where the comma means intersection of the two events. Its marginal sums are $\sum_yp_{X,Y}(x,y)=p_X(x)$ and $\sum_xp_{X,Y}(x,y)=p_Y(y)$, because the corresponding events form disjoint partitions. The variables $X,Y$ are independent precisely when $p_{X,Y}(x,y)=p_X(x)p_Y(y)$ for every pair of values. Under this condition,

$$E[XY]=\sum_x\sum_yxy\,p_{X,Y}(x,y) =\left(\sum_xxp_X(x)\right)\left(\sum_yyp_Y(y)\right)=E[X]E[Y].$$

Define the covariance by $\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]$. Expanding the square and using linearity proves

$$\begin{aligned} \operatorname{Var}(X+Y) &=E[X^2]+2E[XY]+E[Y^2]-(E[X]+E[Y])^2\\ &=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y). \end{aligned}$$

For two random variables with finite second moments, independence is sufficient for variance additivity; more generally, their variances add exactly when their covariance is zero. Expanding the variance of a sum produces a cross term $2(E[XY]-E[X]E[Y])$, which vanishes for independent variables. Identical marginal distributions alone do not remove that term. All sums used in the proof here are finite, so their rearrangement requires no convergence assumption.

A Bernoulli variable $I$ equals one with probability $p\in[0,1]$ and zero with probability $1-p$. Thus $E[I]=1p+0(1-p)=p$. Since $I^2=I$, $\operatorname{Var}(I)=E[I^2]-E[I]^2=p-p^2=p(1-p)$.

Let $n$ be a nonnegative integer and let $I_1,\ldots,I_n$ be Bernoulli variables with the same success probability $p$. For $n\ge1$, mutual independence means that for every $(a_1,\ldots,a_n)\in\{0,1\}^n$,

$$P(I_1=a_1,\ldots,I_n=a_n)=\prod_{j=1}^nP(I_j=a_j).$$

This is joint independence, not merely independence of every pair. Set $X=I_1+\cdots+I_n$. If $n\ge1$ and $0<p<1$, an ordered outcome with $k$ successes has probability $p^k(1-p)^{n-k}$. There are $\binom nk$ distinct choices of its success positions, and those events are disjoint. Therefore

$$P(X=k)=\binom nkp^k(1-p)^{n-k},\quad k=0,\ldots,n.$$

The endpoint cases are stated separately, without evaluating $0^0$: when $p=0$, $P(X=0)=1$; when $p=1$, $P(X=n)=1$; all other values have probability zero. If $n=0$, the empty sum is $X=0$ with probability one, for every $p\in[0,1]$. These are all included in the binomial family.

For $n\ge1$ and $0<p<1$, the displayed probabilities sum to $(p+(1-p))^n=1$ by the binomial theorem; each degenerate case also has total probability one. Linearity gives $E[X]=\sum_jE[I_j]=np$. Mutual independence implies pairwise independence by summing the joint factorization over the other coordinates, each of whose marginal probabilities sums to one. Hence $\operatorname{Cov}(I_i,I_j)=0$ for $i\ne j$. Expanding the squared centered sum now gives

$$\operatorname{Var}(X)=\sum_{j=1}^n\operatorname{Var}(I_j)+2\sum_{1\le i<j\le n}\operatorname{Cov}(I_i,I_j)=np(1-p).$$

The expectation and variance formulas also hold in the endpoint and empty-sum cases, where $X$ is constant. Sampling without replacement generally does not satisfy the mutual-independence assumption.

If $X,Y$ are independent, their joint probabilities factor, giving $E[XY]=E[X]E[Y]$ and $\operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)$.

Worked examples

Reverse conditioning. A factory obtains 30% of parts from source A and 70% from B. Defect probabilities are respectively 4% and 1%. The modeled defect probability is $0.3(0.04)+0.7(0.01)=0.019$. Given a defect, the probability of source A is $0.012/0.019=12/19$, not $0.04$. Conditioning reverses the direction of information and must account for source frequencies.

Without replacement. Draw two balls from a bag containing three red and two blue balls, without replacement. The probability both are red is $(3/5)(2/4)=3/10$. The second conditional probability is $2/4$, not $3/5$, so the two draws are not independent.

Graded practice

1. Foundation. A fair die is rolled. Let $A$ mean even and $B$ mean greater than four. Find $P(A\cup B)$.
2. Core. For two fair independent coin tosses, are the events “first is heads” and “exactly one head” independent? Justify by probabilities.
3. Core. For five independent trials with success probability $1/3$, compute $P(X=2)$, $E[X]$ and $\operatorname{Var}(X)$.
4. Proof. Prove that $\operatorname{Var}(X)=0$ in a finite model if and only if $X=E[X]$ at every outcome of positive probability.

Full solutions

1. $P(A)=3/6$, $P(B)=2/6$ and $P(A\cap B)=1/6$, since only outcome six lies in both. Thus the union probability is $4/6=2/3$.
2. On the four equally likely pairs, both events have probability $1/2$ and their intersection is the single outcome HT, with probability $1/4$. This equals the product, so they are independent. Their verbal relationship alone does not settle independence.
3. $P(X=2)=\binom52(1/3)^2(2/3)^3=80/243$. The expectation is $5/3$ and variance is $5(1/3)(2/3)=10/9$.
4. Variance is the sum of nonnegative terms $(X(\omega)-E[X])^2p(\omega)$. It is zero exactly when every summand is zero. For $p(\omega)>0$, this is equivalent to $X(\omega)=E[X]$; zero-probability outcomes contribute nothing. Conversely, that condition makes every summand zero.

Summary and term definitions

An event is a set of outcomes; a conditional probability renormalizes within a positive-probability event. Independence is a factorization property, not mutual exclusion. Expectation is a probability-weighted mean; variance is the mean squared deviation. State the random experiment, equal-likelihood assumptions and replacement policy before counting. Conditional probabilities with zero-probability conditioning events are not defined by the finite ratio formula.