Why measure a linear relationship?
Knowing two variances does not tell us how two quantities move together. Covariance records whether their centered values tend to have the same sign, but its units change when we measure height in centimeters instead of meters. Correlation removes those units. It is useful for describing linear association, not as a complete description of the joint law.
Suppose and are real random variables with finite second moments. Write , , and . Their covariance is . The product is integrable by Cauchy–Schwarz. If both variances are positive, define
A constant variable has variance zero, so its correlation is undefined, not zero. Rescaling by a positive constant leaves correlation unchanged; a negative rescaling reverses its sign. Adding constants does neither.
Theorem 32.1.1
(The correlation bound and its equality cases)
For square-integrable , . If both variances are positive, . Moreover exactly when almost surely for some nonzero real ; the sign of is the sign of .
Proof
(A nonnegative square proves both the bound and equality)
Set and . If , then almost surely and the inequality is immediate. Otherwise, for every real ,
At the right side is , so . With positive variances equality holds precisely when the nonnegative square has expectation zero, hence almost surely. Conversely such a relation with nonzero slope gives equality. The phrase “almost surely” allows exceptional outcomes of probability zero.
The best affine prediction
We now predict after observing , restricting ourselves to a line . The loss is its mean squared error . This is a population problem: the distribution and its moments are assumed known; it is not a claim about a line fitted to a finite data set.
Theorem 32.1.2
(Least-squares line)
If have finite second moments and , the unique pair minimizing the mean squared error is
The minimum error is . When it is . If is constant almost surely, all lines with give the same best constant prediction, and their coefficient pairs are not unique.
Proof
(Complete the square, without assuming normality)
Write . Then , with . Expanding gives
For fixed the unique minimum in is zero. Completing the remaining square yields
Positive makes the minimizing unique, namely , and then fixes . The value of the minimum is the first two terms; when it equals . If , then almost surely, the error is for every , and every pair with is optimal.
Proof X-ray
Two separate squares, one for each coefficient
The key move is the change of variables from to . In the new coordinates the error splits into , which only the intercept controls, plus a quadratic in alone. Choosing the intercept is therefore the same as matching means, and the slope problem is the same quadratic that appeared in the correlation bound.
The hypothesis is used exactly once, to make the quadratic in strictly convex and its minimizer unique. Finite second moments make every expectation in the expansion finite. Nothing about the shape of the joint law is used, which is why no normality is needed. The optimality conditions say that the residual has mean zero and covariance zero with : these are consequences of the optimum, not assumptions of independence.
The best prediction among all square-integrable functions of is , as established in the conditional-expectation note. The line above minimizes over a smaller class, so its error cannot be smaller. It equals the conditional-mean predictor only when that conditional mean is affine in almost surely. The value is the fraction of the population variance of removed by the best affine predictor with intercept; it does not measure causation, accuracy for each observation, or the predictive value of arbitrary nonlinear functions.
Concept lensGeometric
Correlation as a cosine, the line as a projection
On square-integrable random variables, is an inner product once almost-sure equal variables are identified, with norm . For the centered variables and , the definitions read
So is the cosine of the angle between and , and the correlation bound is the Cauchy–Schwarz inequality, with equality exactly when the vectors are parallel. The best slope is the coefficient of the orthogonal projection of onto the line spanned by ; the residual is the perpendicular component, and the minimum error is Pythagoras. Uncorrelated means orthogonal after centering, which is a statement about one angle, not about the whole joint law.
The worked examples below use the formulas and in three different regimes.
Worked example 32.1.3
(A noisy measurement)
Let have mean and variance . Let be independent of , with mean zero and variance , and set . Then , and . The best line is , with error . Its correlation is . Independence of the noise gives , so in this example the best line also achieves the best error among all functions of . Zero covariance with would suffice for the line calculation, but by itself would not establish this conditional mean.
Worked example 32.1.4
(Predictions shrink toward the mean in standard units)
Let , , and , so . The best line is
An observation , two standard deviations above its mean, gives the prediction , only one standard deviation of above its mean. In general : in standard units the prediction is the observed deviation multiplied by . The minimum error is .
Worked example 32.1.5
(An edge case: a predictor without variation)
Let almost surely, and let have mean and variance . Then , so is undefined and the formula for does not apply. For any line, almost surely, and
which is minimized exactly when . The pairs and are both optimal and give the same prediction , with error . A constant predictor carries no information, so the best one can do is predict the mean of .
Counterexample mode
Zero correlation can hide complete nonlinear dependence
Let be uniform on and put . Then , , , , and . Thus and the best affine prediction is the constant , with error . Yet is determined by : the nonlinear predictor has zero error. Zero correlation excludes linear association in this sense; it does not imply independence or unpredictability.
Common mistakes
Common mistake
(Inverting the line instead of regressing in the other direction)
In the noisy-measurement example, the best line predicting from has slope . The best line predicting from is not obtained by solving for , which would give slope . Its slope is . The two slopes multiply to , not to one; the lines coincide only when .
Common mistake
(Reading the strength of association from the slope)
The slope has units of per unit of ; the correlation has none. Measuring in hundredths multiplies by and leaves unchanged. In the noisy-measurement example the slope is and ; replacing the noise variance by keeps the slope but lowers to .
Exercises
Quick check
(Let be uniform on and . Compute and the least-squares line predicting from .)
Use the equality case of the correlation bound.
Solution · Answer
Here and . The relation has negative nonzero slope, so . Indeed , and . The line is exactly and has zero error.
Quick check
(Let be Bernoulli with , and , where is independent of with mean zero and variance . Find , the optimal line and its error.)
Independence makes .
Solution · Answer
, and independence gives . The slope is and the intercept is . Its residual is , so its error is ; equivalently .
Quick check
(For the nonlinear counterexample, find . Compare its error with that of the best affine line. Does prove independence?)
is a function of .
Solution · Answer
Since , and its error is zero. The best affine line is the constant and has error . The variables are dependent: for instance, but . So does not prove independence.
Quick check
(Suppose , and . Can these be moments of any pair of square-integrable variables? Explain.)
Compare with .
Solution · Answer
No. The covariance bound requires , which violates. Equivalently, the claimed correlation would be .