Motivation
A distribution name should summarize a mechanism. Counting successes in a fixed number of independent trials differs from waiting for a fixed number of successes. Sampling without replacement differs from independent sampling. A good model starts by identifying those distinctions; choosing a familiar formula first can hide the dependence that matters most.
This note develops a connected collection of discrete and continuous laws. Generating functions will do more than store formulas: multiplication will encode independent addition, and differentiation will recover moments. The continuous analogues connect waiting times, integrals and changes of variable.
Definitions
Definition 4.1.1
(Probability and moment generating functions)
For a nonnegative integer-valued variable , its probability generating function (PGF) is
At zero use , so . The power series also converges absolutely for . Its coefficients recover the probabilities. For a real variable , the moment generating function (MGF) is wherever that expectation is finite. We say that an MGF exists near zero if it is finite for every in some open interval containing zero. Existence of a mean alone does not guarantee this stronger condition.
Definition 4.1.2
(Bernoulli, binomial, Poisson and waiting-time counts)
A Bernoulli variable with parameter takes values zero and one with probabilities . A binomial variable counts successes in independent such trials. A Poisson variable , , has masses for . The exponential series verifies that they sum to one. We also allow parameter zero, defined as a point mass at zero.
For , let . A geometric failure count has , . The trial number of the first success is instead . A negative binomial failure count records failures before the th success, where is a positive integer. These support conventions must be stated before using a moment formula.
Main results
Theorem 4.1.3
(PGFs encode addition and factorial moments)
If are independent nonnegative integer-valued variables, then for . Moreover,
where either side may be infinite. If both are finite, .
Proof
(Differentiate inside the interval, then take the limit)
Independence of and makes the bounded variables and independent, so . The power series has radius of convergence at least one, so for it may be differentiated termwise:
Every term is nonnegative and increases with . As , monotone convergence gives the limits and , finite or infinite. Finally gives the variance formula. These are one-sided limits, not a promise that a PGF extends analytically beyond one.
Proof X-ray
Where independence and monotonicity enter
Only the product rule uses independence, and only through . Dependent counts with the same marginals can give the sum a different PGF: if is a fair bit, then , while . The moment identities use no independence. Their key move is the order of two limits: differentiate strictly inside the unit interval, where the series converges, and only then send to one. Monotone convergence licenses that passage because the coefficients and are nonnegative; the same fact makes the identities hold with value when a moment is infinite. At no point is the series evaluated at or beyond .
When an MGF is finite on an open interval around zero, its derivatives at zero give ordinary moments: . To justify differentiation, choose with . On a smaller interval, polynomial factors times are bounded by a constant times . This integrable bound permits repeated differentiation under the expectation.
For the mechanism yields . Expanding gives , while differentiating gives and . Hence . For a Poisson variable,
Multiplication shows that sums of independent Poisson variables are Poisson with summed parameters. Equal mean and variance alone do not characterize the Poisson law; this conclusion comes from the entire generating function.
For the geometric failure count, summing a geometric series gives . Failures between successive successes are independent geometric counts, so
The coefficient is : among the first trials choose the earlier successes, and require a success on the final trial. The total trial count is , with mean and unchanged variance.
Theorem 4.1.4
(A density under a one-to-one transformation)
Suppose has density on an open interval , and is a continuously differentiable bijection with nonzero derivative. Then has density
and zero outside . Endpoints of probability zero do not change the law. If there are several smooth inverse branches, sum their contributions where applicable.
Proof
(Change of variable on an interval)
Because is continuous and never zero on the interval , it has constant sign, so is strictly monotone, is an open interval, and is continuously differentiable with . For an interval , . Substitute . If increases this gives ; if decreases the limits reverse and . Both cases give . Intervals determine the law, so this is the density of , and takes no values outside . The absolute value records interval length when the transformation reverses orientation; the support is part of the answer.
Proof sketch or proof idea
From exponential waiting to gamma sums
An exponential variable with rate has density on . Integrating its tail gives for , so its continuous waiting-time law is memoryless. For , direct integration gives
The gamma density with shape and rate is
The substitution normalizes the density. Integration by parts gives , since vanishes at both endpoints. Consequently
Replacing by inside the integral yields for . Independent gamma variables with the same rate add their shapes. Here one may justify identification either by convolution or by the standard uniqueness theorem for MGFs finite near zero; that uniqueness theorem is used, not proved, in this note. In particular, the sum of independent rate- exponential waiting times has gamma shape .
Concept lensAnalytic
A gamma tail is a Poisson sum
For an integer shape , the gamma law with rate also has a tail formula:
Both sides equal one at , and differentiating the sum telescopes to , which is minus the gamma density. When the exponential waiting times are gaps between arrivals, this reads: the th arrival comes after time exactly when at most arrivals occur by time , a Poisson count with mean .
The normal law and its square
The standard normal density is . Its normalization can be derived by squaring the Gaussian integral and using polar coordinates. Completing the square gives
Thus and . For with , , so its mean and variance are . This parameterization makes the second parameter a variance when the law is written .
For , both inverse branches and contribute. For ,
This is gamma shape , rate , also called chi-square with one degree of freedom. Independent squared standard normals add to gamma shape , rate , giving mean and variance . Omitting the negative inverse branch would leave a function integrating to .
Rare independent successes approach Poisson counts
Theorem 4.1.5
(Poisson limit of rare successes)
Let and, for , let . For every fixed integer , ; consequently converges in distribution to .
Proof
(Three factors with known limits)
Write the binomial mass as a product of three factors:
The empty product for is one. The first factor tends to one, the exponential limit handles the second power, and the last factor tends to one. At any fixed noninteger threshold the distribution function is a finite sum of these masses, proving convergence in distribution to a Poisson law. This approximation concerns many independent opportunities with individually small probabilities and a stable total mean; a large trial count by itself does not justify it.
Worked examples
Worked example 4.1.6
(Sampling without replacement needs a different variance)
A population has objects, exactly marked; draw objects uniformly without replacement. The marked count has mass for feasible . Put . Each draw indicator has mean . For two different positions, , so their covariance is when . Summing gives
With , the mean is and variance . Independence would incorrectly give variance . For the count is deterministic and the displayed fraction is unnecessary.
Worked example 4.1.7
(Thinning a Poisson count)
Given , retain each item independently with probability , independently of , and call the retained count . Conditional on , is binomial, so
Hence . This derivation states both independent mechanisms. Arbitrarily retaining the first half of a random batch would not satisfy the conditional binomial assumption.
Worked example 4.1.8
(Inverse transforms produce exponential waiting times)
Let be uniform on and set . For , . Thus is exponential with rate . Its mean is , not . More generally is uniform on ; integrating and gives mean and variance .
Worked example 4.1.9
(Two different waiting questions)
With independent success probability , the number of failures before the third success has mean and variance . The number of trials through that success has mean and the same variance. Its probability of equaling five is ; the fifth trial must succeed, leaving two successes in the first four.
Common mistakes
Common mistake
(A parameter is meaningless without its convention)
A gamma scale equals the reciprocal of its rate . The means and agree after conversion. A geometric failure count starts at zero, while a trial count starts at one. Write the density or mass and support whenever conventions could differ.
Common mistake
(Dropping the absolute value or the support)
Let be exponential with rate and . Here maps onto and decreases, with and . Without the absolute value the formula gives , a negative “density”. With it, on and zero elsewhere, so is uniform; writing without the support would not even integrate to one. This is the inverse-transform example run backwards.
An MGF can fail to be finite for every positive despite all ordinary moments being finite; the lognormal law is an example. Its moments alone therefore do not authorize the near-zero MGF arguments used above. A characteristic function always exists, since its integrand has modulus one, and is a different tool.
Normal sums require independence or an appropriate joint Gaussian model; a named marginal law says nothing by itself about the relation between two variables.
Counterexample mode
Normal marginals do not make a normal sum
Claim: if and are each normal, then is normal. Let , and put when and when . Since has the same law as and the event is unchanged by the sign flip,
so . But on and otherwise. Hence and , which lies strictly between zero and one. A normal law with positive variance has no atom, and the degenerate law at zero would need this probability to be one; so is not normal and the claim fails. Sums of independent, or jointly Gaussian, normal variables are normal.
Summary
Counting mechanisms produce binomial, hypergeometric, Poisson and negative binomial laws with different dependence and support assumptions. PGFs explain independent addition and factorial moments. Continuous transformations connect uniform, exponential, gamma and normal laws; their Jacobian factors and support restrictions are indispensable parts of a distribution calculation.
Exercises
Quick check
(Let be Poisson with mean and let . Find the PGF, mean and variance of . Is Poisson?)
Solution · Answer
. Adding a constant gives and . It is not Poisson: it assigns no mass to zero, whereas every Poisson law with finite positive mean does. The unequal mean and variance also rules it out.
Quick check
(Independent trials succeed with probability . Find the mean and variance of the trial number of the second success, and the probability that it equals four.)
Solution · Answer
The failure count has mean and variance . Add two to get trial-count mean ; variance stays . For the second success on trial four, choose its one earlier success among the first three: .
Quick check
(If is exponential with rate , find the density and mean of .)
Solution · Answer
The inverse is on , with derivative . Therefore there and zero elsewhere. Using the gamma moment integral, . The real-power moment follows by the same substitution as the integer-moment calculation, and is finite here.
Quick check
(Let be independent gamma variables with shapes and common rate . Find the law, mean and variance of their sum. Explain what breaks when the rates differ.)
Solution · Answer
Multiplying MGFs gives , so the sum is gamma shape , rate , with mean and variance . Different rates produce factors with different denominators and generally no single gamma law. Independence remains necessary for multiplication.
Quick check
(From objects, three marked, sample four without replacement. Find the mean and variance of the marked count, explaining the covariance sign.)
Solution · Answer
Here , so and . Seeing a marked object removes one from the remaining population, lowering the next marked probability; the negative covariance is the reason for the finite-population correction.