Motivation
A learning system may estimate a distribution very accurately and still make an unsuitable decision. Suppose a detector must choose between two classes, one common and one rare. Maximizing the probability density of the observation under each class ignores how often the classes occur. It also ignores whether the two kinds of mistake have different consequences. Before selecting an algorithm, we therefore need to distinguish the object being estimated from the action being evaluated.
There is a second distinction. A low dimensional representation can preserve the directions in which observations vary most, or preserve the directions that distinguish labeled classes. These are different mathematical objectives. Singular value decomposition addresses approximation of a matrix; Fisher discrimination addresses separation relative to within-class scatter. Neither objective automatically solves the other problem.
This unit develops these distinctions from probability and Euclidean geometry. The examples use small matrices so that every computation can be inspected. The setting throughout is finite dimensional, with real observations and explicitly stated sampling assumptions. The notes cover selected statistical foundations, not every estimation method or a complete machine learning course.
Definitions
Definition 1.1.1
(Risk and the Bayes decision)
Let a random pair have . A classifier maps observations to actions in . For a specified nonnegative loss , its population risk is , assuming this expectation exists. Conditional risk at an observed is . A Bayes action minimizes this conditional risk, for almost every .
Under zero-one loss, conditional risk is one minus the posterior probability of the chosen class. The minimum error rule therefore chooses the largest posterior probability. If class priors are and class conditional densities are , the rule compares with . A rule that merely compares with has silently imposed equal priors and equal error costs. This classification use of maximum likelihood should also be distinguished from estimating an unknown parameter by maximum likelihood.
Definition 1.1.2
(Likelihood and parametric estimation)
For independent observations from densities , the likelihood is . A maximum likelihood estimate is a maximizer over a specified parameter space, if a maximizer exists. A parametric model fixes a finite dimensional parameterization before seeing the sample. A nonparametric procedure allows its effective complexity to grow with sample size; it is not a procedure with no assumptions.
For instance, a Gaussian model estimates a mean and variance regardless of . A kernel density estimate on the real line takes the form
where and . Each observation contributes a translated bump. A smaller bandwidth resolves finer features but makes the estimate depend more strongly on the particular sample. Normalization follows by substitution in each integral; accuracy requires additional smoothness and sampling assumptions. A density value can exceed one, since it is probabilities of sets, obtained by integration, that must lie between zero and one.
For regression, write for a design matrix, for responses, and for coefficients. Here the letter denotes a matrix rather than the random observation above; context will make the distinction clear. The squared residual objective is . An intercept, when wanted, is a column of ones. It is not added automatically by the mathematics.
It is useful to distinguish three levels of uncertainty. At the first level, a population distribution defines the posterior probabilities and the ideal decision rule. At the second, a finite training sample is used to estimate that distribution or the rule itself. At the third, a future observation is classified using the fitted rule. The Bayes rule is optimal for the first level, with its specified distribution and loss. Substituting estimated probabilities gives a practical rule, but does not transfer that optimality theorem without further analysis of estimation error.
A related distinction concerns what a regression coefficient means. A coefficient describes variation within the chosen linear representation. Adding a correlated predictor or changing the scale of a column can change its numerical value. The fitted vector belongs to the column space, so different bases for the same space can produce identical predictions with different coefficients. Interpretation therefore needs the design, units and conditioning variables, as well as the number printed by the algorithm. Causal claims require additional assumptions about how observations arise; minimizing squared residuals alone establishes no direction of cause. These considerations do not weaken linear methods. They specify the questions their algebra can answer.
Theorem/Proposition
Theorem 1.1.3
(Bayes rule with unequal error costs)
Suppose correct decisions have zero loss, a false positive costs , and a false negative costs . Wherever both class densities and the posterior are defined, a minimum risk rule chooses class one if
when . Equality permits either action. When a denominator vanishes, compare and directly.
Theorem 1.1.4
(Least squares is orthogonal projection)
For every real design matrix , fitted least squares vectors are exactly the orthogonal projection of onto the column space of . A coefficient vector minimizes the objective if and only if . If has full column rank, the unique coefficient vector is . Without full column rank the fitted vector is still unique, but coefficients need not be.
The likelihood interpretation has extra assumptions. If and, conditional on , the errors are independent with common known , maximizing the conditional likelihood of is equivalent to minimizing the squared residual. Least squares itself remains a valid geometric optimization problem without Gaussian errors. The conditional zero-mean Gaussian model explains a particular statistical interpretation; it is not needed to prove the normal equations.
Theorem 1.1.5
(Truncated SVD gives a best low rank approximation)
Let have singular values and singular vectors . For an integer , the matrix minimizes among matrices of rank at most , with squared error . Repeated singular values at the cutoff can make the minimizer nonunique.
Proof sketch or proof idea
Proof
(Conditional risk and normal equations)
The risks of actions one and zero are respectively and . Bayes' formula gives a common positive denominator, so comparing risks yields the stated threshold. Minimizing pointwise minimizes their expectation, by the tower property.
For least squares, let be the orthogonal projection of onto the column space. Every candidate is in that space, and is perpendicular to it. Pythagoras gives . Thus the optimum has , exactly when its residual is orthogonal to every column. Full column rank makes positive definite, since for , establishing the coefficient formula and uniqueness.
Proof
(Why discarding the smallest singular directions is optimal)
Let be the column space of a candidate , of dimension at most , and let be its orthogonal projector. Orthogonal decomposition of each column gives . In an orthonormal eigenbasis of , this trace is , with and , including zero eigenvalues when necessary. Its largest possible value is at most : moving available weight toward a larger eigenvalue never decreases the sum. The truncated SVD attains this bound. Its omitted orthogonal rank-one terms have squared Frobenius norms , giving the error formula.
Proof X-ray
What the projection proof actually uses
The proof uses a linear subspace and the Euclidean inner product. It does not use random sampling, unbiased noise, or a population regression model. Conversely, claims about future prediction error need information about how new data are generated. An algebraically optimal fit to one matrix is not by itself a statistical guarantee.
For centered observations stored as rows of , the sample covariance is when . Its eigenvectors are the right singular vectors of . Principal component analysis chooses directions that preserve variance, or equivalently minimize squared reconstruction error after centering. Using an uncentered matrix generally answers a different question because the mean contributes to the singular directions.
Concept lensGeometric
Retained variance and reconstruction error are one quantity
Let the rows of the centered matrix be projected by an orthogonal projector onto a -dimensional subspace. Pythagoras applied to each row gives , and summing over gives
The left side is the squared reconstruction error. The subtracted sum is times the total variance of the projected points, because the rows are centered and so are their projections. Since does not depend on , maximizing retained variance and minimizing reconstruction error select the same subspaces. Read as spread, the criterion asks for the directions in which the data vary most; read as distance, it asks for the subspace closest to all observations. Both descriptions concern the same data and the same objective, and neither refers to the class labels.
Fisher's two-class criterion instead starts with class means and a specified within-class scatter matrix . Its normalization must be fixed before reporting numerical quotients. One population convention is ; an unnormalized sample scatter uses another convention. Multiplying the whole matrix by a positive constant preserves the maximizing direction but rescales the quotient. Put . If is positive definite and , maximize
Writing turns the quotient into . Cauchy–Schwarz bounds it by , attained exactly when is a nonzero multiple of . This is a complete derivation under positive definiteness. If sample size is too small for to be invertible, one must specify a restriction or regularization; writing an ordinary inverse is invalid.
Worked examples
Worked example 1.1.6
(A likely observation can belong to the common class)
Suppose , , and priors are , . Comparing likelihoods alone selects class one. But the unnormalized posterior weights are and , so the posterior of class one is . Under zero-one loss, select class zero. If a false negative costs ten times a false positive, the conditional risks become for action zero and for action one. The appropriate decision changes to one. The probability model did not change; the objective did.
Worked example 1.1.7
(Fitting a line and checking its residual)
Take inputs and responses . With intercept first, the columns of are and . Thus and . The fitted intercept and slope are and . Predictions are , giving residual . Its sum is zero and its inner product with the input column is zero, verifying both normal equations. The residual sum of squares is . A zero sum of residuals follows from the intercept column; it is not a general feature of every regression design.
Worked example 1.1.8
(Reconstruction and discrimination can choose perpendicular directions)
For , a best rank-one approximation is , with squared error one. This is a statement about reconstructing . For a separate classification model, suppose and . Fisher's direction is proportional to , and the maximum quotient is two. If the two classes have equal priors and common within-class covariance with means and , the total covariance is . Its largest-variance direction is the second coordinate, which carries no mean separation. Retaining only that principal component discards the feature that Fisher's criterion selects.
Common mistakes
Counterexample mode
More variance need not mean more class information
Consider the claim that the leading principal component is always the best single feature for telling the classes apart. It fails in the last example: the second coordinate has the same Gaussian distribution in both classes if we further assume Gaussian class conditionals with the stated common covariance. Observing it alone leaves the posterior equal to the prior. Its large variance records variation unrelated to the label. Labeling every large principal component as an informative classifier feature confuses two objectives.
Common mistake
(An optimizer is not automatically unique or statistically consistent)
A design with duplicate columns has multiple least squares coefficient vectors. A density family may have no likelihood maximizer on its parameter space. A kernel estimate with an extremely small bandwidth can faithfully reproduce sample spikes without estimating the underlying density well. Always state what exists, what is unique, and what depends on a sampling limit. A training objective alone answers none of these questions automatically.
Summary
Minimum error classification compares posteriors, and unequal costs modify the threshold. Likelihood estimation selects parameters within an assumed family. Least squares projects responses onto a column space. SVD chooses a representation for reconstruction, while Fisher's criterion measures mean separation relative to within-class variation. These connections are useful precisely because their assumptions and objectives remain distinguishable.
Exercises
Quick check
(Priors are 0.8 and 0.2 for classes zero and one. A false positive costs 3 and a false negative costs 2. Derive the likelihood-ratio threshold and decide when the ratio is 5.)
Compare conditional risks before substituting numbers.
Solution · Answer
The threshold is . Ratio five is below six, so select zero. Equivalently, the ratio of the risk of choosing one to the risk of choosing zero is .
Quick check
(Let the design matrix have two identical columns, both (1,1) transposed, and response (1,3) transposed. Find every least squares coefficient vector and the fitted response.)
The fit depends on the sum of the two coefficients.
Solution · Answer
Writing , minimize . Thus every vector is a minimizer, with fitted response and residual . The minimum squared error is two. Inverting the singular Gram matrix would miss the actual issue.
Quick check
(Show that the kernel density estimator integrates to one. What fails if the kernel has integral two?)
Use a separate change of variable for each observation.
Solution · Answer
For each summand let , so . Its integral, including the factor , is . Summing gives one. If , the estimator integrates to two and is not a probability density unless renormalized. Nonnegativity is also necessary for a density.
Quick check
(Find a best rank-one approximation to diag(2,2) and its error. Explain why the answer is not unique.)
Compare retaining either coordinate direction.
Solution · Answer
is optimal and has Frobenius error two, or squared error four. So is . More generally works for any unit vector : the orthogonal direction is discarded with singular value two. The theorem guarantees an optimal error, not a unique basis at a repeated singular value.