Motivation
Saying that random variables “approach a limit” is incomplete until the meaning of approach is specified. A sample average can settle along almost every infinite sample path. A standardized fluctuation can instead approach a nondegenerate distribution while continuing to vary. These statements involve different scales and different modes of convergence.
The earlier weak-law calculation used only the variance of one average. A strong law must control an entire infinite sequence of averages. The missing tool is a maximal inequality: it controls the chance that any partial sum in a block becomes large. For the central limit theorem, characteristic functions turn independent addition into multiplication and reveal why only the first two moments survive the normalization.
Definitions
Definition 6.1.1
(Modes of convergence)
Let be real random variables. Almost-sure convergence, written almost surely, means that the set of outcomes on which the numerical sequence converges has probability one. Convergence in probability means
Convergence in mean square, or , means . These three definitions compare variables on the same probability space. Convergence in distribution, , means at every continuity point of . It concerns only their laws, so the variables need not initially be realized on the same space. At a jump of convergence is not required.
Definition 6.1.2
(Characteristic function)
The characteristic function of a real variable is , , with . It always exists because , even when has no mean. If are independent, . Scaling gives .
Main results
Theorem 6.1.3
(Some implications, with their scope)
Almost-sure convergence implies convergence in probability, and convergence implies convergence in probability. Convergence in probability implies convergence in distribution. If the distributional limit is a constant , then convergence in distribution to also implies convergence in probability to . The converse implications for general nonconstant limits are false.
Proof
(Comparing events and distribution functions)
For the first assertion, the events decrease to a probability-zero set under almost-sure convergence. Continuity from above gives , bounding each later deviation probability. The implication is Markov's inequality applied to the squared error.
For the distributional implication, for ,
First let tend to infinity and then let decrease to zero at a continuity point . If the limit is constant, the distribution functions at and tend to zero and one, respectively, forcing .
Theorem 6.1.4
(Finite-variance strong law)
Let be independent and identically distributed with and . Then
The proof below uses finite variance. A stronger integrable iid strong law exists, but is not proved by the argument below.
Theorem 6.1.5
(Iid finite-variance central limit theorem)
Under the preceding hypotheses with ,
The proof below derives the characteristic-function limit completely, then invokes the explicitly stated Lévy continuity theorem to pass from characteristic functions to distributions.
Proof sketch or proof idea
A maximal inequality controls the whole path
Lemma 6.1.6
(Kolmogorov's maximal inequality)
Let be independent, centered variables with finite variances, and . Then for every ,
Proof
(First crossing times)
Let be independent, centered variables with finite variances, and . For , let be the event that is the first index with . These events are disjoint. Future increments are independent of and , and have mean zero. Therefore
Sum over . The events are disjoint with union , and outside that union, so .
Proof
(Dyadic blocks and Borel–Cantelli)
For , apply the maximal inequality with and . The resulting probabilities are at most and have a finite sum over . The first Borel–Cantelli lemma says that events with summable probabilities occur only finitely often almost surely. Its proof is short: the probability of their union after index is at most the tail sum, which tends to zero.
Thus, almost surely, for all sufficiently large the maximum up to is smaller than . For , it follows that . Intersect the probability-one events for to conclude almost surely. This proves the strong law. A separate probability-one set for each fixed tolerance is insufficient until the countable intersection is taken.
Proof X-ray
Why checking isolated sample sizes is insufficient
Chebyshev at a single index gives , whose sum over diverges, so Borel–Cantelli cannot be applied index by index. Restricting to dyadic indices makes the bounds summable, but convergence along alone would say nothing about the indices in between. The key move is that the maximal inequality pays the same price for every partial sum up to at once, so the dyadic bound also fills the gaps, at the cost of the factor in . Independence enters only in the maximal inequality, where the future increment must be uncorrelated with ; finite variance makes every bound finite; identical distribution supplies the common . The argument closes with the countable intersection over , which turns “eventually within , for each fixed ” into convergence on one probability-one set.
The local characteristic-function expansion
Proof
(Second-order expansion and the product)
Let , so and . Write
where as . One can choose bounded on all real : Taylor's formula handles , and division by bounds each remaining term for . Consequently is dominated by a constant times and tends pointwise to zero as . Dominated convergence gives
Only a finite second moment was used. A third moment or an unjustified infinite Taylor series is unnecessary. For independent copies and a fixed real ,
For complex factors this limit follows by using the logarithm near one: there, so multiplication of the logarithm by tends to .
Proof X-ray
What each hypothesis does in the central limit argument
Finite variance is used once, to dominate by a constant times ; it yields the expansion with an error and nothing more. Centering and standardization remove the linear term and fix the quadratic coefficient at . Independence and identical distribution turn the characteristic function of the sum into the th power of one function. The key move is the scaling : it makes the quadratic term exactly of size , so the th power has a finite nonzero limit, while the error vanishes after multiplication by . Lévy's continuity theorem then converts this pointwise limit into convergence in distribution.
The theorem that closes the CLT argument
The standard normal characteristic function is . One direct verification differentiates its integral under the sign of integration, justified by integrability of . Integration by parts using gives , with , whose solution is the claimed function.
We now use Lévy's continuity theorem: if characteristic functions converge pointwise to a function continuous at zero, that function is the characteristic function of a probability law and the corresponding laws converge in distribution to that law. In particular, pointwise convergence to the characteristic function of a known variable implies convergence in distribution to it. The theorem's proof, which uses tightness and uniqueness of characteristic functions, is external to this note. Applying it to the displayed limit proves the CLT. This identifies the exact bridge between the algebraic calculation and a statement about probabilities.
A vanishing perturbation preserves a distributional limit
Suppose and in probability on the same space. For any , the distribution function of is bounded below by and above by . Choose to be continuity points, let grow, and then let shrink at a continuity point of the limit. This proves . The result explains why sufficiently small approximation errors can be ignored at the final scale. It also warns against replacing the true mean by an estimate without checking that scale: a centering error tending to zero need not remain negligible after multiplication by .
Worked examples
Worked example 6.1.7
(Almost sure does not imply mean square)
Let be uniform on and set . For every outcome with , the indicator eventually vanishes, so almost surely. Nevertheless , which diverges. Rare values can grow rapidly enough to defeat mean-square convergence despite pathwise eventual disappearance.
Worked example 6.1.8
(In probability can allow infinitely many deviations)
Take independent events with for , and put . Then in probability. For each fixed ,
Thus the event of no further successes after has probability zero. Taking a countable union over shows that successes occur infinitely often almost surely, so there is no almost-sure convergence to zero. Independence is used in the product; the divergent sum alone would not suffice.
Counterexample mode
Equal laws do not force close outcomes
Let be a fair sign and . Every has exactly the distribution of , so . At odd , however, . Hence convergence in distribution need not give convergence in probability to the variable on that same space. A distributional limit does not specify how the variables are coupled.
Worked example 6.1.9
(Averages settle while standardized errors fluctuate)
For iid Bernoulli trials with , the strong law gives almost surely. The CLT simultaneously gives . There is no contradiction: the normal scale magnifies the shrinking error by . With , a fixed- normal approximation uses standard deviation for the average. The CLT itself supplies no numerical error guarantee at that sample size.
Common mistakes
Common mistake
(Almost surely is a statement about infinite paths)
It does not mean that the average equals its mean after some finite index, nor that every conceivable outcome converges. A probability-zero exceptional set is allowed. Convergence in probability permits rare deviations at each index without ruling out infinitely many deviations along an individual path.
A normal approximation with finite-variance iid data is an asymptotic statement. It has no universal convergence rate under those hypotheses alone. Infinite-variance variables, dependent samples, and changing distributions require separate results. If , the variables equal their mean almost surely and the standardized expression divides by zero; use the trivial constant conclusion instead.
Common mistake
(Convergence in distribution does not carry moments)
Convergence in distribution does not in general imply convergence of means or variances. The growing-spike example has laws approaching a point mass at zero but its first moment remains one. Additional integrability control is needed to pass an expectation through a limit.
Summary
Modes of convergence answer distinct questions about paths, errors and laws. A maximal inequality plus summable dyadic bounds proves the finite-variance strong law for entire sample paths. The CLT follows from a second-order characteristic-function expansion, independence, and an explicitly external continuity theorem. Neither proof licenses conclusions outside its hypotheses or a finite-sample error estimate it does not provide.
Exercises
Quick check
(Let for a single uniform . Prove almost-sure and convergence to zero. Compare with the independent-event example.)
Solution · Answer
At every outcome , eventually , so all later values vanish. Thus convergence is almost sure. Also , proving mean-square convergence. The events here are nested and highly dependent; the independent-event product argument cannot be applied to them, despite identical individual probabilities.
Quick check
(For centered independent variables with common variance four, bound using the maximal inequality.)
Solution · Answer
. The bound is . It bounds the probability of any threshold crossing in the entire interval, not only a deviation at time 100.
Quick check
(Suppose and all variables are on one probability space. Prove convergence in probability, and explain why the fair-sign counterexample does not contradict this.)
Solution · Answer
For , the points are continuity points of the point-mass-at-zero distribution. Therefore . The fair-sign limit is nonconstant, so the special constant-limit implication does not apply.
Quick check
(For iid variables of mean and variance , express as a standardized probability and find its limit.)
Solution · Answer
The event is . Since the standard normal distribution function is continuous at two, its probability tends to . No numerical accuracy at any particular follows without further information.
Quick check
(If has mean zero and finite variance , derive the characteristic-function expansion and the limit of the characteristic function of , for independent copies. Include .)
Solution · Answer
The same bounded-remainder argument gives . Independence gives the th power at , tending to . For this is ; for , a centered zero-variance variable equals zero almost surely, and the limiting characteristic function is identically one, the point mass at zero.