Probability depth: hidden conditions, protocol and dependence

A sample space $\Omega$ lists possible outcomes of a modeled experiment. An event is a subset. A probability assignment satisfies $P(E)\ge0$, $P(\Omega)=1$, and additivity for disjoint events; in a general probability model this extends to countable disjoint unions. For a finite sample space it is enough to assign nonnegative probabilities to outcomes summing to one and add them over events.

Consequently $P(E^c)=1-P(E)$ and

$$P(E\cup F)=P(E)+P(F)-P(E\cap F).$$

To derive the second identity, split $E\cup F$ into disjoint pieces $E\setminus F$, $F\setminus E$, and $E\cap F$, then count the overlap only once.

Uniform counting, $P(E)=|E|/|\Omega|$, is valid only if the finite elementary outcomes are equally likely. A short list of named categories does not make them equally likely.

The multiplication principle counts sequential choices: if stage one has $m$ options and each has $n$ stage-two options, there are $mn$ ordered choices. Selecting $k$ distinct objects in order from $n$ gives $n(n-1)\cdots(n-k+1)=n!/(n-k)!$. Ignoring order divides by $k!$, because each selected $k$-element subset has exactly $k!$ orderings. Hence

$$\binom nk=\frac{n!}{k!(n-k)!},\qquad0\le k\le n.$$

For $P(B)>0$, define $P(A\mid B)=P(A\cap B)/P(B)$. It renormalizes the possible outcomes after restricting to event $B$. It follows that $P(A\cap B)=P(A\mid B)P(B)$.

Events $A,B$ are independent when $P(A\cap B)=P(A)P(B)$. When $P(B)>0$, this is equivalent to $P(A\mid B)=P(A)$. Disjoint events with positive probabilities are not independent: their intersection is empty while the product of their probabilities is positive.

If $B_1,\ldots,B_m$ partition the sample space with positive probabilities, then

$$P(A)=\sum_iP(A\mid B_i)P(B_i).$$

This is the law of total probability, obtained by adding the disjoint pieces $A\cap B_i$. If $P(A)>0$, Bayes' formula is

$$P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{\sum_iP(A\mid B_i)P(B_i)}.$$

A random variable $X$ assigns a real number to each outcome. In this finite model, its distribution is $p_X(x)=P(X=x)$ and

$$E[X]=\sum_xxp_X(x)=\sum_{\omega\in\Omega}X(\omega)p(\omega),\qquad \operatorname{Var}(X)=E[(X-E[X])^2].$$

The two expressions for the expectation agree by grouping outcomes with the same $X$ value. For real constants $a,b$, distributing the finite outcome sum gives

$$E[aX+bY]=\sum_{\omega}(aX(\omega)+bY(\omega))p(\omega)=aE[X]+bE[Y].$$

Thus linearity needs no independence. With $\mu=E[X]$ and $E[1]=1$, expanding the square gives

$$\operatorname{Var}(X)=E[X^2]-2\mu E[X]+\mu^2E[1]=E[X^2]-E[X]^2.$$

Define the joint distribution by $p_{X,Y}(x,y)=P(X=x,Y=y)$, where the comma means intersection of the two events. Its marginal sums are $\sum_yp_{X,Y}(x,y)=p_X(x)$ and $\sum_xp_{X,Y}(x,y)=p_Y(y)$, because the corresponding events form disjoint partitions. The variables $X,Y$ are independent precisely when $p_{X,Y}(x,y)=p_X(x)p_Y(y)$ for every pair of values. Under this condition,

$$E[XY]=\sum_x\sum_yxy\,p_{X,Y}(x,y) =\left(\sum_xxp_X(x)\right)\left(\sum_yyp_Y(y)\right)=E[X]E[Y].$$

Define the covariance by $\operatorname{Cov}(X,Y)=E[XY]-E[X]E[Y]$. Expanding the square and using linearity proves

$$\begin{aligned} \operatorname{Var}(X+Y) &=E[X^2]+2E[XY]+E[Y^2]-(E[X]+E[Y])^2\\ &=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y). \end{aligned}$$

For two random variables with finite second moments, independence is sufficient for variance additivity; more generally, their variances add exactly when their covariance is zero. Expanding the variance of a sum produces a cross term $2(E[XY]-E[X]E[Y])$, which vanishes for independent variables. Identical marginal distributions alone do not remove that term. All sums used in the proof here are finite, so their rearrangement requires no convergence assumption.

A Bernoulli variable $I$ equals one with probability $p\in[0,1]$ and zero with probability $1-p$. Thus $E[I]=1p+0(1-p)=p$. Since $I^2=I$, $\operatorname{Var}(I)=E[I^2]-E[I]^2=p-p^2=p(1-p)$.

Let $n$ be a nonnegative integer and let $I_1,\ldots,I_n$ be Bernoulli variables with the same success probability $p$. For $n\ge1$, mutual independence means that for every $(a_1,\ldots,a_n)\in\{0,1\}^n$,

$$P(I_1=a_1,\ldots,I_n=a_n)=\prod_{j=1}^nP(I_j=a_j).$$

This is joint independence, not merely independence of every pair. Set $X=I_1+\cdots+I_n$. If $n\ge1$ and $0<p<1$, an ordered outcome with $k$ successes has probability $p^k(1-p)^{n-k}$. There are $\binom nk$ distinct choices of its success positions, and those events are disjoint. Therefore

$$P(X=k)=\binom nkp^k(1-p)^{n-k},\quad k=0,\ldots,n.$$

The endpoint cases are stated separately, without evaluating $0^0$: when $p=0$, $P(X=0)=1$; when $p=1$, $P(X=n)=1$; all other values have probability zero. If $n=0$, the empty sum is $X=0$ with probability one, for every $p\in[0,1]$. These are all included in the binomial family.

For $n\ge1$ and $0<p<1$, the displayed probabilities sum to $(p+(1-p))^n=1$ by the binomial theorem; each degenerate case also has total probability one. Linearity gives $E[X]=\sum_jE[I_j]=np$. Mutual independence implies pairwise independence by summing the joint factorization over the other coordinates, each of whose marginal probabilities sums to one. Hence $\operatorname{Cov}(I_i,I_j)=0$ for $i\ne j$. Expanding the squared centered sum now gives

$$\operatorname{Var}(X)=\sum_{j=1}^n\operatorname{Var}(I_j)+2\sum_{1\le i<j\le n}\operatorname{Cov}(I_i,I_j)=np(1-p).$$

The expectation and variance formulas also hold in the endpoint and empty-sum cases, where $X$ is constant. Sampling without replacement generally does not satisfy the mutual-independence assumption.

Probability models and information

Correct fraction arithmetic can still answer the wrong probability question if the experiment, observation procedure or independence assumptions are missing. We use hypothetical finite models to examine hidden conditions, pairwise versus mutual independence, and a host protocol whose details affect an observed posterior. Prerequisites are finite sample spaces, conditional probability, Bayes' formula, expectation and basic counting.

1. Sample-space descriptions

If $P(B)=0$, the finite conditional-probability ratio is undefined, not automatically zero.

The experiment must specify which outcomes are equally likely. A uniformly selected unordered subset has a different sample description from an ordered sequence, but either is valid if probabilities are assigned consistently. Counting ordered outcomes in the numerator and unordered outcomes in the denominator mixes two sample spaces and is invalid even when both counts are individually correct.

2. Conditional independence need not survive mixing

Choose a hidden coin type $T$ with equal probabilities: type H has head probability $3/4$ and type L has head probability $1/4$. After choosing one type, toss that same coin twice independently conditional on its type. The conditional assumption is explicit; it is not independence after forgetting the type.

Let $A$ mean first toss heads and $B$ mean second toss heads. Total probability gives $P(A)=P(B)=1/2$. But

$$P(A\cap B)=\frac12\left(\frac34\right)^2+\frac12\left(\frac14\right)^2=\frac5{16},$$

not $1/4$. The events are positively associated in the mixed model. A head makes the high-head type more plausible: $P(T=H\mid A)=3/4$, so

$$P(B\mid A)=\frac34\frac34+\frac14\frac14=\frac58>\frac12.$$

The second toss has not been physically altered by the first; the information about the shared hidden type has changed.

For two equally likely types with head probabilities $p,q$, the same argument gives

$$P(A\cap B)-P(A)P(B)=\frac{p^2+q^2}{2}-\left(\frac{p+q}{2}\right)^2=\frac{(p-q)^2}{4}.$$

Thus equal type biases remove this source of dependence, while unequal biases create it. Replacing each conditional probability by a marginal average and then multiplying would erase exactly this term.

3. Pairwise is weaker than mutual independence

Events $A_1,\ldots,A_m$ are mutually independent if for every nonempty set of indices $S\subseteq\{1,\ldots,m\}$,

$$P\left(\bigcap_{j\in S}A_j\right)=\prod_{j\in S}P(A_j).$$

Pairwise independence imposes this condition only for two-element sets $S$. For three events, the three pair conditions and the triple-intersection condition are all required; the singleton conditions are automatic.

Toss two independent fair bits $X,Y$, with four equally likely outcomes. Let $A=\{X=0\}$, $B=\{Y=0\}$ and $C=\{X=Y\}$. Each has probability $1/2$. Each pair has exactly one common outcome, so every pairwise intersection has probability $1/4$, equal to the product. Yet

$$P(A\cap B\cap C)=P((X,Y)=(0,0))=\frac14\ne\frac18.$$

They are pairwise independent but not mutually independent. Mutual independence justifies the three-event product because its definition includes the triple-intersection condition; pairwise independence alone does not justify that product.

Conditioning on $C$ leaves only outcomes $(0,0)$ and $(1,1)$, each with conditional probability $1/2$. In that conditional model, $A$ and $B$ always occur together, so $P(A\cap B\mid C)=1/2$, while $P(A\mid C)P(B\mid C)=1/4$. Originally independent events can become dependent after selecting a shared condition. The new sample space, not just the notation, has changed.

4. An observation protocol is part of the likelihood

There are three closed boxes, one prize placed uniformly at random. A player first selects box one. A host knows the prize location, always opens an unselected nonprize box, and then offers a switch to the other unopened box. Without conditioning on which box the host opens, switching wins exactly when the player's first choice was wrong, hence with probability $2/3$. This argument does not require the host's tie-breaking to be uniform, only the stated knowledge and always-open/always-offer protocol.

Now suppose the player observes that the host opened box three. Let $J$ denote prize location, and let $\theta\in[0,1]$ be the probability the host opens box three when $J=1$, the case where either unselected box is available. Then

$$P(O_3\mid J=1)=\theta,\quad P(O_3\mid J=2)=1,\quad P(O_3\mid J=3)=0.$$

Bayes' rule gives

$$P(J=2\mid O_3)=\frac{1/3}{\theta/3+1/3}=\frac1{1+\theta}.$$

Uniform tie-breaking means $\theta=1/2$ and gives $2/3$. Always choosing the larger-numbered available box gives $\theta=1$ and posterior $1/2$. Never choosing box three when both are available gives $\theta=0$ and posterior one after that observation. The observation is more or less informative under different protocols, even though the unconditional switch success stays $2/3$.

There is no contradiction: averaging the conditional outcomes under the corresponding protocol recovers the same unconditional chance. A purported solution that says “one losing box was opened, so the two remaining boxes must be equally likely” supplies neither the relevant likelihoods nor a valid symmetry argument. The host used knowledge to choose the observation, breaking that naive symmetry.

5. Odds expose likelihood versus prior belief

Let $B$ be a hypothesis with $0<P(B)<1$, and let observed event $A$ have positive conditional probabilities under both $B$ and $B^c$. Applying the conditional definition twice yields

$$\frac{P(B\mid A)}{P(B^c\mid A)} =\frac{P(A\cap B)}{P(A\cap B^c)} =\frac{P(A\mid B)}{P(A\mid B^c)}\frac{P(B)}{P(B^c)}.$$

The first factor is a likelihood ratio; the second is prior odds. Their product is posterior odds. Evidence can favor a previously unlikely hypothesis without making it more likely than its alternative. Specifically a likelihood ratio greater than one raises its probability, but whether its posterior exceeds one half also depends on the prior odds. This is an algebraic statement about a specified model, not a guarantee about the truth of an actual hypothesis.

6. Sampling without replacement: expectation does not require independence

Let $N,K,n$ be integers with $N\ge0$, $0\le K\le N$, and $0\le n\le N$. Draw $n$ objects sequentially without replacement from $N$ distinct objects, of which $K$ are marked; at each step choose uniformly among all remaining objects. Every ordered sample of distinct objects then has the same probability. Forgetting order gives the uniform distribution on $n$-element subsets, since each subset has exactly $n!$ orders. Let $X$ count marked selections. For each of the $n$ draw positions, define indicator $I_j=1$ for a marked object and zero otherwise. By symmetry each object is equally likely at a fixed position, so each indicator has marginal probability $K/N$ when $N>0$. The indicators need not be independent; in degenerate cases such as $K=0$ or $K=N$, they are constant. For $N>0$, linearity gives

$$E[X]=\sum_{j=1}^n E[I_j]=\frac{nK}{N}.$$

When $N=0$, necessarily $K=n=0$ and $X=0$, so $E[X]=0$. When $N=1$, the permitted draws give a deterministic count, hence $\operatorname{Var}(X)=0$. No multiplication of joint probabilities was used, so independence is unnecessary. In contrast, variance of a sum includes covariance terms unless independence or another zero-covariance condition is established. The binomial model from independent replacement trials cannot simply be imposed on this experiment.

For $N>1$ and $n\ge2$, the ordered objects at any two distinct draw positions are uniform among the $N(N-1)$ ordered pairs of distinct objects. Exactly $K(K-1)$ such pairs are both marked. Therefore $E[I_iI_j]=P(I_i=I_j=1)=K(K-1)/(N(N-1))$. If $n<2$, there are no such pairs and the covariance sum is empty. Subtracting $(K/N)^2$ gives covariance $-p(1-p)/(N-1)$ with $p=K/N$. Thus

$$\operatorname{Var}(X)=np(1-p)\frac{N-n}{N-1}.$$

The finite-population factor makes the variance zero when every object is drawn, as it must: then $X=K$ deterministically. This provides a model-consistency check beyond algebraic substitution.

Graded synthesis and diagnostics

1. Logical distinction. Prove that two disjoint events with positive probabilities are not independent. Give a finite example of independent events with a nonempty intersection.
2. Hidden mixture. Derive the dependence gap $(p-q)^2/4$ for equally likely hidden types, then identify exactly when the two head events are independent in that model.
3. Conditional selection. In the two-bit example, compute $P(A\mid C)$, $P(B\mid C)$ and $P(A\cap B\mid C)$, and diagnose the claim that independence is always preserved by conditioning.
4. Host protocol. Derive the observed-box-three switch posterior for general $\theta$. Explain how it can vary while the unconditional switching success remains $2/3$.
5. Evidence. Prove that a likelihood ratio greater than one raises $P(B)$ under the positive-probability assumptions above. Does it force $P(B\mid A)>1/2$?
6. Sampling. Derive the variance formula for without-replacement sampling from the indicator pair probabilities. Treat $N=1$ separately rather than dividing by $N-1$.

Full solutions

1. Disjointness gives intersection probability zero, whereas positive marginal probabilities have a positive product; independence would equate those unequal numbers. For a nonempty independent example, on the four fair two-bit outcomes take $A=\{X=0\}$ and $B=\{Y=0\}$. Their marginals are $1/2$ and their intersection is the single outcome $(0,0)$, with probability $1/4$.
2. Conditional independence gives joint probability $(p^2+q^2)/2$, while each marginal is $(p+q)/2$. Their difference is $[2p^2+2q^2-(p+q)^2]/4=(p-q)^2/4$. The events are independent exactly when this gap is zero, namely $p=q$, including boundary biases. Conditional independence given type held throughout; this conclusion concerns the model after mixing.
3. Given $C$, the two surviving outcomes are equally likely. Each marginal is $1/2$ and the intersection has conditional probability $1/2$, not the product $1/4$. Thus independence is not generally preserved by conditioning. In this selected model the two events coincide, so the changed dependence has an explicit sample-space explanation.
4. The three joint weights for $O_3$ and $J=1,2,3$ are $\theta/3,1/3,0$. Normalizing makes the switch-success posterior $1/(1+\theta)$. Unconditionally a first incorrect choice occurs with probability $2/3$ and the protocol then necessarily leaves the prize in the switch box; a first correct choice makes switching lose. Conditioning on one particular host action selects different mixtures of those initial cases as $\theta$ changes, without altering their original probabilities.
5. Write prior odds $o=P(B)/(1-P(B))>0$ and likelihood ratio $\ell>1$. Posterior odds are $\ell o>o$. The map $u\mapsto u/(1+u)$ is increasing for positive $u$ because cross multiplication preserves the inequality, so posterior probability exceeds prior probability. It need not exceed one half: if $o<1/\ell$, posterior odds remain below one. For instance a sufficiently small positive prior satisfies that inequality for any fixed finite likelihood ratio.
6. Each indicator has variance $p(1-p)$. For distinct positions, covariance is $K(K-1)/(N(N-1))-K^2/N^2=-p(1-p)/(N-1)$. Expanding the squared centered sum gives $np(1-p)+n(n-1)[-p(1-p)/(N-1)]=np(1-p)(N-n)/(N-1)$. For $N=1$, the permitted $n$ is zero or one; $X$ is respectively zero or the deterministic marked count, so variance is zero directly. If $N=0$, only the empty draw is permitted and again $X=0$; formulas involving $K/N$ are not used.

Summary and terms

Conditional independence concerns a model after fixing information; marginal independence concerns it after that information is averaged out. Mutual independence controls joint intersections beyond pairs. A protocol specifies how an observation is selected, hence its likelihood. A covariance records a cross term in variance and need not vanish when expectation is additive. The experiment and information flow must be specified before probability formulas can be applied meaningfully.