Motivation
Two measurements can each have a familiar distribution while their relationship remains unknown. Even knowing both marginals does not determine a probability such as . A joint law supplies the missing relation. Conditioning then asks a different question: how should a prediction change after one measurement is revealed?
For a continuous observation, the event normally has probability zero. Dividing by therefore cannot define its conditional distribution. Densities and conditional expectation provide a precise replacement. We will derive the rules and use them to separate variation within groups from variation between their means.
Definitions
Definition 5.1.1
(Joint density and marginals)
A pair has joint density if for every measurable set . Here and its integral is one. The marginal densities are
Integrals may be written over all real values by setting the density to zero outside its support. Independence holds exactly when almost everywhere. Product-looking algebra on a nonrectangular support does not establish independence.
Definition 5.1.2
(Conditional density)
At a value with , a version of the conditional density is
The qualifier “version” matters: changes at a set of -values having probability zero do not alter any of the defining probability identities. Conditional laws at such exceptional values are not uniquely determined by the joint law.
Definition 5.1.3
(Conditional expectation)
For integrable , a conditional expectation is an integrable function satisfying
for every Borel set . It is determined almost surely. In the density setting wherever the absolute integral is finite. More generally one writes for an integrable -measurable variable satisfying this identity for every event in the information collection , a -field. This definition extends conditioning beyond densities; its general existence theorem is used here without proof.
Main results
Theorem 5.1.4
(Multivariate change of variables)
Let have density on an open region . Suppose is a continuously differentiable bijection with continuously differentiable inverse and nonzero Jacobian determinant. For ,
Regions omitted from or are harmless only if they have probability zero. For finitely many disjoint smooth inverse branches, sum the branch contributions.
The formula follows by applying the ordinary two-dimensional change-of-variables theorem to the integral defining probabilities. A determinant measures local area scaling. Using the forward determinant without taking its reciprocal is a different operation and generally gives an unnormalized answer.
Theorem 5.1.5
(Tower property and variance decomposition)
If is integrable and are -fields, then
In particular . If , define . Then
The nesting of information is essential in the first statement. Arbitrarily conditioning on two unrelated variables in succession does not justify a tower simplification. In the second identity the conditional variance is a random variable depending on the observed group, while its expectation is a number.
Proof sketch or proof idea
Why a conditional density is not division by zero
For bounded measurable and an event , Fubini's theorem gives
At exceptional where the ratio is undefined we may choose any probability density: those choices have zero weight in the outer integral. This integrated identity is what the conditional formula means. It never divides the probability of a singleton by zero. For integrable, unbounded , absolute integrability permits the same calculation.
Conditional expectation as the best square-error prediction
Proof
(Orthogonality and the defining identity)
Put . The defining averaging identity implies for bounded measurable ; extension by truncation gives the same statement for square-integrable . Conditional Cauchy–Schwarz ensures is square-integrable when is. For any square-integrable prediction , expand
The cross term vanishes by the preceding orthogonality. Thus minimizes mean squared error, uniquely up to almost-sure equality. Taking gives total variance. This explains the two terms: unexplained fluctuation around each conditional mean, and fluctuation of the means themselves.
For the tower property, is -measurable. For every , also , so . The defining property and almost-sure uniqueness identify with .
Proof X-ray
Where nesting and orthogonality are used
Two different facts carry the proof. The tower identity needs only the defining property and one inclusion: an event is used as a test event for , which is legitimate because . Without that step has no justification. The variance decomposition rests on orthogonality, : this is the defining identity with the indicator replaced by a bounded function and then, by truncation, by a square-integrable one. The hypothesis is what makes every product in the expansion integrable. The key move is the Pythagorean split , whose second piece is a function of . Choosing the constant closes the argument: by the tower property the first square averages to , and the second is because .
Sum and proportion: the gamma–beta connection
Let be independent gamma variables with shapes and common rate . Define and . The inverse is , with and . Its Jacobian determinant has absolute value . Substitution gives
The first factor integrates to one. Since the whole joint density integrates to one, the second does too. Therefore and are independent, has gamma shape , and has beta parameters . This also proves
Taking ratios of beta integrals yields and . Subtracting the square of the mean gives variance . The common-rate assumption made the exponential depend only on ; unequal rates generally destroy this independence.
The corresponding covariance identity
If both have finite second moments, conditioning on also gives
Here the conditional covariance is . To prove the identity, subtract each overall mean and decompose each centered variable into its residual and its conditional mean minus overall mean. The two mixed terms have expectation zero by conditional orthogonality. The residual product gives the first term, and the product of the conditional means gives the second. Taking recovers total variance. Unlike both terms of the variance identity, covariance terms can be negative; the formula separates sources of linear association without claiming each source increases it.
Worked examples
Worked example 5.1.6
(A triangular support changes the conditional law)
Let on , and zero elsewhere. The triangle has area , so this is normalized. Integrating gives on and on . For , is uniform on , giving and conditional variance .
Now and , so . Total variance gives . Direct integration yields and the same variance . The variables are dependent: knowing changes the possible range of .
Worked example 5.1.7
(A jointly Gaussian conditional distribution)
Suppose is jointly Gaussian with means , positive standard deviations , and correlation . Its joint density has quadratic exponent
Since , division by the marginal normal density of leaves a normal density in . Thus
The conditional mean is linear here because of the joint Gaussian hypothesis. If , the joint density factors and the variables are independent. At the joint law is singular and this two-dimensional density argument does not apply; an exact affine relation replaces the positive conditional variance.
Counterexample mode
Uncorrelated normal marginals can still be dependent
Let be standard normal and let be an independent fair sign. Set . Symmetry makes standard normal, and . Nevertheless surely. For example the events and coincide and have a probability strictly between zero and one, so they are not independent. The pair is not jointly Gaussian. Normal marginals plus zero covariance do not meet the previous example's hypotheses.
Worked example 5.1.8
(Variation in a random Poisson rate)
Let have finite second moment and suppose is Poisson with parameter , interpreting parameter zero as a point mass at zero. Then and . Consequently
A nonconstant random rate creates variance above the mean. The marginal law is generally a mixture, not a Poisson law with parameter . Conditioning makes the calculation simple while retaining the extra variation.
Common mistakes
Common mistake
(A zero conditional mean does not imply independence)
If is uniform on and , symmetry gives almost surely, yet is determined by . Zero conditional mean does not imply independence: still reveals the magnitude of . Conversely, , a nonlinear function.
Common mistake
(A product-looking formula that ignores the support)
The density on is constant on its support, so it can look like a product of two constants. Independence, however, is a statement about the whole density, support included. The marginals and multiply to , which is not identically on the triangle and is positive on the half of the square where . A constant density on a triangle is not the product of two uniform densities on a square.
Common mistake
(Scaling by the forward Jacobian)
Let be independent and uniform on , and . The forward map has Jacobian determinant , the inverse has determinant , and the correct density is on . Multiplying by the forward determinant gives on , which integrates to . The formula uses the inverse map both to evaluate and to scale area. Write the inverse and the transformed region first: a determinant alone cannot identify the new support.
Conditional expectation given more information can change either direction at a particular outcome. Its square-error advantage is an average statement, not a guarantee of smaller error on every sample. Likewise the tower property is an averaging identity, not permission to replace observations by their means inside arbitrary nonlinear functions.
Summary
A joint density records dependence through both its values and its support. Jacobians transform areas and must accompany the inverse map. Conditional densities are justified by integrated identities, while conditional expectation formalizes prediction from information. Tower and total variance follow from those defining identities and orthogonality; Gaussian and gamma–beta examples show when especially simple conditional structures actually occur.
Quick check
(The joint density is on . Verify normalization, find and , and use the tower property to find .)
Solution · Answer
The integral is . Thus on and on . Integrating against this density gives . Since , the tower property gives .
Quick check
(Let be independent standard normals. Set . Find the inverse map, Jacobian, joint density, and determine whether are independent.)
Solution · Answer
The inverse is , with absolute Jacobian . Also , giving on all of . This factors into two densities, so the variables are independent. The conclusion uses equal initial variances as well as the joint Gaussian model.
Quick check
(Let equal or with equal probabilities and be Poisson with rate . Find , , and . Compare with a Poisson variable having the same mean.)
Solution · Answer
and , so and . Total probability gives . A Poisson variable of mean two has variance two and zero probability ; neither matches the mixture.
Quick check
(A jointly Gaussian pair has means , standard deviations and correlation . Find and . Verify total variance.)
Solution · Answer
The conditional mean is . The conditional variance is . The conditional mean as a random variable is , whose variance is . Adding recovers .
Quick check
(If are independent unit-rate exponentials, find the density of and compute without evaluating a double integral.)
Solution · Answer
Both gamma shapes are one, so has beta density, which is uniform on . Since is equivalent to , its probability is . Positivity of almost surely justifies the equivalence.