Motivation
Adding two measurements and sorting a sample are both transformations of several random variables. Neither operation can be understood from a list of marginal distributions alone. A sum depends on how its summands are coupled; an ordered sample is dependent even when the unsorted observations were independent.
The unifying method is to start with a joint law, transform the region as well as the variables, and integrate out coordinates that are no longer wanted. Convolution is one instance of this method. Order-statistic formulas are another: their factorial constants count which labeled observations could occupy the ordered positions. Spacings then show how a change of coordinates can expose either dependence or independence, depending on the original distribution.
Definitions
For real variables with densities , independence means their joint density is almost everywhere. If , the convolution of these densities is the function
The actual integration region is where both factors are nonzero: must belong to the support of and to the support of . Writing an integral over the real line does not remove this support restriction.
For a sample , its order statistics are the sorted values . Assume throughout the order-statistic results that the sample is independent and identically distributed with an absolutely continuous distribution function and density . Ties then have probability zero. The variable is the th smallest observation, not the original observation .
For ordered uniform observations , define the spacings
They are positive and sum to one. The last spacing is the gap to a fixed right endpoint. For positive, unbounded observations such as exponentials, we will instead use the spacings from zero through the largest observation, with no final fixed-endpoint gap.
Main results
Theorem 8.1.1
(Independent sums, differences and ratios)
For independent variables with densities , a density of is . A density of is . Since , the ratio is defined almost surely and has density
Each formula is an almost-everywhere density statement, with the original support restrictions imposed inside its integral. Infinite values at exceptional points do not by themselves invalidate a density.
Theorem 8.1.2
(Ordered iid samples and their marginals)
The full ordered vector has density
For ,
For , the joint density of at is
and is zero for . A factor raised to the zeroth power is interpreted as one.
Theorem 8.1.3
(Uniform spacings and exponential spacings behave differently)
For iid uniform observations on , the first spacings have constant joint density on , , with . All spacings are exchangeable, with
For iid rate- exponential observations, the successive spacings from zero are independent exponentials with rates .
Proof sketch or proof idea
Transform first, then marginalize
Proof
(Sum, ratio and product by change of variables)
For a sum, use coordinates . The inverse is and its absolute Jacobian is one. The joint density of is consequently ; integrating out proves the convolution formula. In the discrete integer-valued setting the same reasoning is a disjoint-event sum: under independence.
For a ratio, use away from . Its inverse is , with absolute Jacobian . The change-of-variables theorem on the two regions and , followed by integration over , proves the ratio formula. Omitting the absolute value would allow a negative density.
Similarly, for a product , the inverse of is , with absolute Jacobian . Thus
These formulas need the joint density. Under dependence, replace by the actual joint density if one exists; a singular coupling may have no two-dimensional density at all.
Sorting counts permutations
Proof
(Ordered densities by counting labeled positions)
Except on the zero-probability tie set, each ordered vector has exactly labeled preimages. Each permutation has Jacobian of absolute value one and the same product density, so adding the contributions proves the full ordered density.
To retain only , integrate the coordinates below over their ordered region. The unordered integral over is , and symmetry splits it equally among the orderings, giving . The upper ordered region similarly contributes . Multiplying by proves the marginal formula. Retaining two coordinates partitions the remaining values into three ordered regions: below , between and , and above . Their integrals supply the three powers and three factorials in the joint formula. Thus the constants count available labeled positions rather than being normalizing guesses.
An independent check uses a count: means at least observations lie below . Its distribution function is the corresponding binomial upper tail. Differentiating at points where gives the same marginal density.
Uniformization and gap geometry
Proof
(Uniform order statistics and spacings)
For absolutely continuous , are iid uniform on . Strict increase is unnecessary. To see the marginal assertion, for take the right endpoint of the set . Continuity and the limits of give distribution value there; a flat segment contributes no probability. Hence . Coordinatewise transformation preserves independence. Ordered probabilities can therefore be studied on the uniform scale, even if conversion back needs a generalized quantile rather than an ordinary inverse.
In particular has beta parameters . Using the beta integral and its recurrence gives
The map from to the first spacings is triangular with determinant one. Its image is the open simplex in the theorem, so the density there is constant. Permuting any of the gap coordinates, including the dependent last coordinate, induces an affine change of the first coordinates with absolute determinant one. Constant density on that simplex therefore proves exchangeability.
Since their sum is one, exchangeability gives the mean of every gap. Also for , and the same tail holds for every gap. Integrating the tail yields and the displayed variance. All off-diagonal covariances agree; expanding the variance of the fixed sum gives , proving the covariance formula.
Concept lensStructural
Uniform spacings as normalized exponential gaps
Let be iid exponential with any common rate, and . Then has the same law as the uniform spacings . Changing coordinates to and the first ratios, with Jacobian , factors the density into a gamma density for times the constant on the simplex. Uniform gaps are thus independent exponential gaps rescaled to total one, and that rescaling is what couples them.
Why exponential spacings really are independent
Proof
(Factorization of exponential spacings)
For ordered exponential observations write , so and . The Jacobian is one. The ordered joint density becomes
The support is now a Cartesian product and each factor is a normalized exponential density. This factorization proves independence, not merely lack of covariance. It also gives by adding the mean spacings.
Proof X-ray
Where the two spacing proofs diverge
Both spacing proofs begin with the same move: a triangular change to gap coordinates with Jacobian one, which carries the ordered density to the gaps without a correction factor. The iid and absolute-continuity hypotheses enter there, through the labeled preimages of each ordered vector.
The proofs then split on two questions that a product-shaped expression cannot answer by itself. The first is the transformed support. For uniforms it is the simplex , which couples every coordinate, so independence is impossible and only exchangeability survives. For exponentials it is the orthant , a Cartesian product. The second is the density. The uniform density is constant, whereas the identity is the key step that turns the exponential density into a product of normalized one-variable factors. Independence needs both the product support and the factorization.
The uniform covariance then closes by one algebraic observation: the total of the gaps is the constant one, so its variance is zero. Unbounded exponential observations have no fixed right endpoint, hence no final gap and no such constraint.
Worked examples
Worked example 8.1.4
(A triangular sum density comes from interval overlap)
Let be independent uniform variables on . For , the integration interval is . Its length is for and for , and zero otherwise. Hence . The shape comes from overlap lengths, not a presumption that sums are uniform.
Worked example 8.1.5
(Uniform ratios and products can have large densities)
With the same independent sample, the ratio integrand is for and zero unless . Therefore for , and for . Its mean is infinite because diverges. Bounded numerator and denominator do not bound a ratio when the denominator can approach zero.
For , the allowed range is when . Thus there. It is unbounded near zero but integrates to one. Independence directly gives , agreeing with the density integral.
Worked example 8.1.6
(A normal ratio produces a Cauchy law)
For independent standard normals , the ratio formula gives
The law is symmetric, but has no expectation: its positive and negative first-moment integrals both diverge. It would be invalid to use symmetry to report mean zero, or to average a conditional mean without checking integrability of the ratio first. A symmetric principal-value integral is not an expectation.
Worked example 8.1.7
(Extremes and a range are jointly constrained)
For uniform observations, the minimum and maximum have joint density for . Set . The inverse coordinates satisfy , , with Jacobian one. Integrating out gives . For three observations the density is and the expected range is . Treating minimum and maximum as independent would miss their ordered support and produce the wrong range law.
Counterexample mode
Identical marginals do not authorize convolution
Let be uniform on and set . Both marginals are uniform, but is uniform on , not triangular. Also almost surely, a point mass rather than the ratio density above. This coupling is concentrated on the diagonal and has no two-dimensional density. The missing ingredient is independence, not an algebraic correction to the convolution formula.
Common mistakes
Common mistake
(Exchangeable spacings are not independent)
Order statistics are not independent observations relabeled harmlessly. Their shared ordering constrains them, and their spacings may share a fixed total. Exchangeability means invariance under permutations of the coordinates; it does not imply independence. The negative uniform-gap covariance measures one consequence of the fixed-sum constraint.
Common mistake
(Uniform spacings do not describe nonuniform data)
A probability integral transform makes the observations uniform on the probability scale, not on the original measurement scale. Uniform-spacing formulas therefore do not automatically describe physical distances between ordered exponential or other nonuniform observations. The separately proved exponential-spacing factorization is a different result.
Common mistake
(Dropping absolute continuity or independence)
Absolute continuity matters. With discrete observations ties have positive probability, so an ordered vector need not have the displayed density or distinct preimages. With dependent continuous observations, the labeled densities need not agree under permutation. Each assumption has a specific job in the proof.
Summary
Convolution and ratio formulas come from joint transformations and marginalization with explicit supports. Sorting introduces permutation factors and ordered integration regions. Uniformization gives beta order-statistic laws, while uniform spacings have a constant simplex density and negative covariance. Exponential spacings instead factor into independent laws with changing rates. These distinctions are consequences of the model and region, rather than interchangeable distribution formulas.
Exercises
Quick check
(For two independent uniforms on , derive the density of and find .)
Solution · Answer
The integrand for the difference is one when and . The overlap has length for , and zero otherwise. Thus .
Quick check
(Independent exponentials have rates two and three. Derive the density, mean and variance of their sum. Explain the common-rate limit of the density.)
Solution · Answer
For , the convolution is , with zero density for . Independence gives mean and variance . In general distinct rates give ; as , the exponential difference quotient tends to , yielding , the shape-two gamma density.
Quick check
(Four iid observations are uniform on . Find the second order statistic's density and mean, and compute by counting.)
Solution · Answer
The formula gives for and mean . The event means at most one of four observations is at most . Its probability is .
Quick check
(For four uniform observations, derive the range density and its expected value from the minimum–maximum joint density.)
Solution · Answer
The minimum–maximum density is on . Set and integrate over to obtain for . Hence .
Quick check
(Four uniform observations create five spacings, including both endpoint gaps. Find the probability that their largest spacing exceeds , and the covariance of two different spacings.)
Solution · Answer
Each spacing exceeds with probability . Two cannot both exceed because the total is one. These five events are disjoint, so the required probability is ; no independence assumption is used. With , the covariance is .
Quick check
(Three iid observations have exponential rate two. Use independent spacings to find the mean and variance of their maximum.)
Solution · Answer
The three independent spacings have rates . Their sum is the maximum, so its mean is and variance is . Independence is what permits the variances to add; it was established from the full transformed density.