Motivation
An experiment may have a complicated outcome, yet a numerical summary such as a count or waiting time may be the object of interest. Distributions describe that summary, while moments and limit theorems answer different questions about its average, spread and large-sample behavior.
Definitions
From outcomes to numbers
Definition 2.1.1
(Random variable and distribution function)
A random variable is a real-valued measurable function on a probability space. It assigns a numerical value to each outcome, and measurability ensures that events such as have probabilities. Its distribution function is for real .
The distribution function describes the probabilities without requiring us to enumerate the underlying outcomes. It is nondecreasing and right-continuous, with limits zero at negative infinity and one at positive infinity.
For a discrete variable, the probability mass function is nonnegative and sums to one over its possible values. A continuous variable with density satisfies . A density can exceed one: it is an area, not the height at one point, that is a probability. Also for each fixed when has a density. A distribution can have both atoms and a continuous part, so these two descriptions are not exhaustive.
Worked example 2.1.2
(An atom and a uniform part)
With probability set ; otherwise draw uniformly on . Then and . Thus , and . No single density represents this whole distribution, because the jump at zero is an atom.
Expectation and variance
Definition 2.1.3
(Expectation)
For a discrete , when the sum is absolutely convergent; for a density, when the absolute integral is finite. In either case is called integrable.
More generally the expectation of is computed by replacing with in the appropriate sum or integral, provided it is integrable. This rule avoids first deriving the distribution of .
If are integrable and are constants, linearity says . Independence is not required. The variance, when , is
The second identity follows by expanding the square and using linearity. Variance is nonnegative by its first expression; a negative numerical result indicates an arithmetic or rounding error. When both and have finite second moments,
where covariance is . Independence and finite second moments imply zero covariance, but zero covariance does not in general imply independence. For example, if is uniform on and , symmetry gives ; nevertheless is determined by .
Worked example 2.1.4
(A count without enumerating its distribution)
In four independent trials, success has probability . Let be the indicator of success on trial , and . Since , and . Linearity and independence yield and . The expectation formula would remain valid without independence; the variance would then need covariance terms.
Common mistake
(Subtracting variances for a difference)
For independent with finite variances, , because and . Writing instead can even give a negative number. For two independent fair bits, the difference takes with probabilities , so its variance is , not .
Proof sketch or proof idea
The proof below derives a probability bound by comparing a nonnegative random variable with an indicator. This simple comparison is enough to establish a weak law under finite-variance independence.
Concentration and averages
Theorem 2.1.5
(Markov inequality)
For a nonnegative random variable with finite expectation and , .
Proof
(Comparison with an indicator)
Because , the inequality holds at every outcome: on the right side equals , and elsewhere it equals . Expectation preserves inequalities, so . Divide by .
Counterexample mode
Markov's inequality needs a nonnegative variable
Claim: for every integrable and . Let take the values and with probability each, and . Then , so the claimed bound would give , whereas . The proof breaks at the outcome , where would say . The repair is to apply the inequality to a nonnegative function of : here , which is true.
Applying Markov's inequality to the nonnegative variable with the threshold , where and , gives Chebyshev's inequality:
Theorem 2.1.6
(Weak law of large numbers with finite variance)
Let be independent and identically distributed with mean and finite variance , and let . Then for every , as .
Proof
(Chebyshev applied to the sample average)
By linearity, . Expanding the square of the centered sum , every cross term has expectation for by independence, so
Chebyshev's inequality applied to then gives
Proof X-ray
Where each hypothesis enters
Finite variance is what makes Chebyshev available: Markov's inequality is applied to the nonnegative variable , whose expectation is . Independence is used exactly once, to delete the cross terms; pairwise uncorrelated variables with common mean and variance would give the same bound. Identical distribution only supplies the common and . The key move is that the variance of the average decays like while stays fixed. The argument closes with an explicit bound, so it also gives a sample size: guarantees a deviation probability at most . If every equals one nonconstant , the cross terms do not vanish, for all , and no decay occurs.
The sample average therefore converges in probability to the mean. It says nothing that makes a particular finite sample equal to the mean, and it does not say the sum stays near the mean.
The central limit theorem
Under the same independent, identical distribution and finite-variance assumptions, with , the central limit theorem states that for each real ,
where is the standard normal distribution function. This is convergence in distribution, a different assertion from convergence in probability to a constant. The theorem is stated here without proof; its proof needs additional tools such as characteristic functions. It gives a limiting approximation, not an exact normal law for each or a universal bound on approximation error.
Worked example 2.1.7
(A normal approximation and its scope)
For independent Bernoulli trials with , has mean and standard deviation . The theorem approximates by as grows. At a finite the mass at the threshold and discreteness matter; a continuity correction may improve an approximation but is not an exact identity.
A distribution for a waiting time
A useful discrete model counts trials until the first success. If trials are independent and have success probability with , the waiting time takes values with
The first trials must fail and the next must succeed. These probabilities sum to one by the geometric series; for the first trial succeeds surely. The expected waiting time is . One derivation uses the tail-sum identity for nonnegative integer-valued , and then sums .
Concept lensAnalytic
Expectation as a sum of tail probabilities
For with values in , write and exchange the order of summation:
The exchange is legitimate because every term is nonnegative; both sides may equal . This description reads the mean from the tail rather than from the mass function. For the geometric waiting time the first trials fail with probability , so with no differentiation of a series.
Theorem 2.1.8
(Memoryless property of the geometric law)
For nonnegative integers and , .
Proof
(A ratio of tails)
The event means that the first trials fail, so , which is positive for . Since ,
Past failures do not alter the probability of a further run of failures because the trials are independent; this conclusion would fail if the success probability changed after each failure.
Worked example 2.1.9
(Waiting for a success)
With success probability , the chance the first success arrives on trial three is . The expected waiting time is four trials. Given failure on the first two, the chance of at least two more failures is still . Expectation is an average over many repetitions, not a promise of success on trial four.
Common mistake
(Convergence does not repair missing assumptions)
Averages of perfectly dependent variables need not concentrate: if every equals the same nonconstant , then for every . Identical marginal distributions alone do not justify the independent-sample calculation.
Joint distributions and covariance
A joint distribution records probabilities for several random variables together. For discrete , write . The marginal mass of is obtained by summing over all possible : . The analogous sum over gives the marginal of . Both sums are needed because knowing the separate marginals does not tell us how the variables are coupled. Independence is the stronger statement for every pair of values.
Worked example 2.1.10
(Equal marginals, different joint laws)
Let both variables be fair bits. In one model they are independent, so all four pairs have probability and . In a second model take ; only and occur, each with probability , so . Both models give each bit the same marginal distribution. A marginal table alone therefore cannot determine their agreement probability.
For finite second moments, covariance measures a particular linear relation. Expanding and using linearity gives . If are independent, their joint distribution factors, so whenever the product is integrable. In the discrete finite case this is a direct factorization of the double sum; the general statement follows from the corresponding integration theorem. As the earlier square example shows, covariance zero is a weaker statement than independence: nonlinear dependence may remain.
The formula for a sum extends to variables:
It follows by expanding the square of the centered sum. Pairwise zero covariance suffices to remove the cross terms; full mutual independence is stronger than needed for this particular calculation. However, that observation alone does not establish the central limit theorem, whose hypotheses here include independent identically distributed variables.
Conditioning as an averaging tool
Suppose form a finite partition with , and is integrable. Define as the expectation under the conditional probability given . Then
For a finite-valued this follows by writing the left side as and applying the law of total probability to each . Interchanging the finite sums gives the formula. The integrable general case follows by the same decomposition of the integral. The weights are the probabilities of the partition cells, not necessarily equal weights.
Worked example 2.1.11
(A mixture of two production lines)
A line is selected with probabilities and . Its expected number of defects per batch is respectively two and six. The overall expected count is . It would be wrong to average the two conditional means as unless the two lines were equally likely. The answer does not require the full conditional distributions.
Conditioning is especially useful when the conditional distribution is simpler than the unconditional one. It does not say that is the observed value of : it is an average over all outcomes within the cell. The distinction matters whenever the cell still contains variability.
Reading a probability bound
Chebyshev's inequality can be loose. For a fair Bernoulli sample of size 100, the bound in the exercise guarantees that the deviation event cannot have probability greater than under the model, but it does not claim that this upper bound is attained. The exact probability could be evaluated by summing binomial masses. Bounds are useful when one wants a guarantee using only a few known quantities; an approximation such as the central limit theorem uses a different argument and has a different meaning.
The finite-variance weak law proved here does not cover every form of the law of large numbers. Stronger results can relax some assumptions or strengthen the mode of convergence, but those claims need their own hypotheses and proofs. Equally, the normal approximation should not be applied mechanically to a heavily dependent sample or to a distribution with infinite variance on the strength of the displayed theorem.
Transformation and the distribution function
A transformation need not preserve the order of values. If , one cannot simply write : negative values of also contribute, and values below must be excluded. Instead, for ,
For this probability is zero. If is uniform on , the length of the interval divided by two yields for , with for . Differentiating on gives . Although the density is unbounded near zero, its integral over is one, so it defines a valid density. The value assigned to the density at the single point zero does not change its probabilities.
The expectation rule gives a shorter way to calculate one feature: . Integrating over gives the same answer. Deriving the entire transformed distribution is necessary for arbitrary probability questions about , but it is unnecessary when the expectation alone is requested.
Common mistake
(Moving a nonlinear function outside the expectation)
In this example , so , while . In general for nonlinear ; the expectation rule substitutes inside the sum or integral. For the gap is exactly the variance, which is why it is never negative.
A distribution function also records point masses through its jumps. At any real , , where is the left limit. To see this, the increasing events have union ; continuity of probability from below makes their probabilities tend to . Subtracting from leaves the event . Thus a continuous distribution function has no atoms, although it need not have a density. This distinction prevents a common overgeneralization from density examples.
The distinction between a distribution and a density also matters at endpoints: changing a density at one point leaves every integral unchanged, but changing the mass assigned to an atom changes the distribution itself. Always specify which description is being used before differentiating or summing.
Summary
A distribution specifies probabilities; expectation and variance describe selected features of it. Linearity of expectation needs integrability but not independence. The weak law concerns concentration of averages, whereas the central limit theorem describes their rescaled fluctuations.
Exercises
Quick check
(A variable takes with probabilities . Find its mean and variance.)
Solution · Answer
and ; hence the variance is .
Quick check
(For 100 independent Bernoulli trials with success probability , use Chebyshev to bound the probability that the sample proportion differs from by at least .)
Solution · Answer
Each trial has variance , so the sample proportion has variance . Chebyshev gives the bound . This is a bound, not the exact probability.
Quick check
(A random variable has density for and zero elsewhere. Find and .)
Solution · Answer
Integration gives and .