Suppose two labeled groups of points have different means. Projecting in the direction between the means seems a natural way to distinguish them. But a mean difference of one unit is not equally useful in every direction. If the points spread very widely in that direction, their projected groups may still overlap heavily. A direction with a smaller mean difference can be more informative when its within-group spread is much smaller.
Fisher's criterion compares these two quantities. It chooses a one-dimensional linear projection with a large squared mean separation relative to within-class scatter. We will derive its optimizer rather than treat an inverse-matrix formula as a rule to memorize. Then we will explain a separate step: turning the projection into a classifier. Maximizing a ratio alone does not specify a threshold, posterior probability or guarantee about future classification error.
State the data and projection conventions
We have two nonempty labeled samples, classes 0 and 1. A data point is a column vector in . Let index the points of class , let , and define the sample class means and their difference by
A nonzero vector assigns scalar score . The projected class means differ by . To measure within-class spread, subtract each point's own class mean, not the overall sample mean.
Definition 19.1.1
(Within-class scatter and the Fisher quotient)
The within-class scatter matrix is
For a direction with , its Fisher quotient is
The denominator equals the sum of squared projected residuals about the two projected class means. It is not the sum of squared distances from a shared overall mean.
The scatter matrix is symmetric and positive semidefinite: for every vector , its quadratic form is a sum of squares . It need not be invertible. The main theorem will explicitly assume positive definiteness.
Our convention uses an unnormalized residual sum. Dividing the whole matrix by the positive constant , when that denominator is positive, gives the usual pooled covariance estimate. Such a common scalar changes the value of but not its maximizing directions. Reweighting the two class scatters differently changes the criterion and may change its direction; that is a different choice of model, not just a notation change.
Derive the direction using an inner product
For a nonzero scalar , the numerator and denominator of both acquire the factor . Hence only the line spanned by is determined by the criterion. A sign also disappears under the square. If we want class 1 to have the larger projected mean, we must choose the sign afterwards.
Theorem 19.1.2
(The Fisher direction for positive-definite scatter)
Assume is positive definite and . Then
Choosing makes . In practice, obtain one such direction by solving rather than forming an explicit inverse.
Proof
(Use the scatter matrix to define the right geometry)
Because is symmetric positive definite, is an inner product. Put . Then , and
The Cauchy–Schwarz inequality in this inner product gives
Divide by the positive denominator. Since , this proves the upper bound. Equality in Cauchy–Schwarz holds exactly when the nonzero vectors and are scalar multiples. Those directions attain the bound. Finally, for with , we have .
Proof X-ray
Where positive definiteness and Cauchy–Schwarz enter
The non-obvious move is to rewrite the numerator as with , so that numerator and denominator are measured in the same geometry; after that the argument is a single application of Cauchy–Schwarz. Positive definiteness of is used twice: it makes a genuine inner product (the equality case needs definiteness), and it makes exist, so that is defined, with because . The condition makes the maximum positive and the maximizers a single line; with every direction gives . If is only semidefinite, the inequality still holds, but need not exist and a direction can have zero denominator, which the section on singular scatter treats by example.
The formula does not simply favor a coordinate because its raw mean difference is large. It adjusts for the whole scatter matrix, including correlations. With diagonal scatter, each coordinate's mean difference is divided by its own within-class scatter. With correlated coordinates, the inverse also accounts for directions in which coordinates vary together.
Concept lensGeometric
Whiten the scatter, then take the mean difference
Put , and . Then , and , so . In whitened coordinates the within-class scatter is the identity, and is times the squared cosine of the angle between and . It is largest exactly when is parallel to , that is, .
If while remains positive definite, every nonzero direction has , and the theorem's uniqueness-up-to-scale statement does not apply. The last worked example of the next section shows how this happens.
Compute a complete two-dimensional example
Worked example 19.1.3
(Account for vertical spread before choosing the projection)
Consider these eight labeled points:
| Class 0 | Class 1 |
|---|---|
Their means are and , so . In either class, the residuals about its own mean are , , , . The class scatter is therefore , and
We may multiply this direction by 16 and use . Its projected class means are 0 and 17. The squared mean difference is ; the within-class scatter after projection is . Hence , agreeing with the theorem.
Using the raw mean difference instead gives numerator and denominator , so . The Fisher direction has a larger quotient because the vertical residual spread is larger and deserves less weight in this criterion.
A possible classifier for assigns class 1 when , the midpoint of the projected class means. To make a total rule, assign equality to class 0. This threshold correctly labels all eight displayed training points. That observation is about this finite sample, not a proof of generalization.
For a new point , the Fisher score is 8, below , so this rule assigns class 0. Projection along with its own midpoint threshold gives score , assigning class 1. The point has no supplied true label: the disagreement illustrates different boundaries, not proof that one of these predictions is correct.
Worked example 19.1.4
(Correlated scatter changes the direction)
Let class 0 consist of and class 1 of . The means are and , so . The class 0 residuals are the points themselves and contribute ; the class 1 residuals contribute . Hence
For we get and , so . The raw direction gives numerator and denominator , so . The reason is that class 0 varies only along ; the direction is orthogonal to that variation, so both class 0 points project to and only class 1 contributes within-class spread.
The projected class means are and , so the midpoint threshold is . The scores are for class 0 and for class 1. The rule "predict class 1 when " labels three of the four training points correctly; the point lies exactly on the threshold and, with equality assigned to class 0, is misclassified.
Worked example 19.1.5
(Equal means: the quotient is blind)
Take one-dimensional classes with sample values and . Both means are , so . The scatter is , and for every : every direction is optimal with value , so the criterion cannot prefer any of them. The classes nevertheless have different spreads, and the rule "predict class 1 when " separates them perfectly. That rule is not a threshold on any score , because on the line the class 1 values lie on both sides of the class 0 values. A criterion built from mean separation does not detect this kind of difference.
A direction does not choose a probability model or threshold
The Fisher quotient contains no threshold. Even after choosing the sign of , we still need a decision rule such as “predict class 1 when .” We might choose using validation data, an application-specific cost, or a probability model. These choices answer different questions from maximizing the training scatter ratio.
Proposition 19.1.6
(Bayes rule for two Gaussians with a shared covariance)
Suppose that, conditionally on class , a point has the Gaussian distribution with the same positive-definite covariance for both classes, that the class priors are with , and that the two kinds of misclassification have equal cost. Put . Then the rule minimizing the error probability predicts class 1 exactly when
Proof
(From the posterior log-odds to a linear rule)
Write out the two Gaussian exponents and expand them. The shared quadratic term in cancels, and the log posterior odds are
Here we used . The Bayes rule with equal costs predicts class 1 when the posterior odds exceed , that is, when the log odds are positive. Rearranging gives the stated inequality.
Proof X-ray
What each assumption buys
Shared covariance: it cancels the quadratic term, so the boundary is linear. With different covariances the log odds keep the term and the boundary is quadratic. Equal costs: they place the threshold at odds ; unequal costs would add a constant to it. Priors: they contribute , and for equal priors the threshold is the midpoint of the two projected means. Scale: is the specific vector . A Fisher direction is defined only up to a scalar and the threshold must scale with it, so an arbitrarily normalized Fisher vector cannot be inserted into the formula while leaving the prior term unchanged. When is proportional to , the vector is parallel to the sample Fisher direction, which is the connection between the two.
This argument explains a connection, not an extra assumption hidden inside the Fisher theorem. The quotient optimization itself does not require Gaussian samples. Conversely, outside the shared-covariance Gaussian model, its linear midpoint rule need not minimize classification error. A larger training quotient is not a universal ordering of classifiers by test accuracy.
Common mistake
(A maximal quotient is not a zero-error classifier)
The quotient measures the separation of projected means relative to scatter; it does not minimize training error, and it does not choose a threshold. In the correlated example above, the direction with the largest quotient still leaves one training point exactly on the midpoint threshold, where the tie convention misclassifies it. Reversing the sign of without reversing the inequality swaps the predicted labels while leaving the quotient unchanged.
Singular scatter contains two different situations
If there are too few independent residual directions, is singular. For example, the residuals within class sum to zero, so the rank of the pooled scatter is at most . In dimensions above that number it cannot be positive definite. Collinear features can also cause singularity even with many observations.
Counterexample mode
A zero-spread separating direction is not handled by blindly taking a pseudoinverse
Take and . Along , the projected means differ but within-class scatter is zero. The quotient as defined has zero denominator and is not a finite real number.
For with , the denominator is and the numerator is 1. Thus the quotient is unbounded as . No direction in the positive-denominator domain attains a finite maximum. The pseudoinverse gives , which is not even a permitted nonzero direction and misses this zero-spread separation.
If instead with the same scatter, every has quotient 1. The null direction changes neither projected mean difference nor within-class spread. There are many optimal directions that are not scalar multiples. Here the pseudoinverse chooses one representative, but the positive-definite theorem's uniqueness statement is still unavailable.
These examples can arise from actual samples, not merely arbitrary matrices: class 0 points and class 1 points produce the first scatter and mean difference. Moving the second class's center to instead produces the second case.
One deliberate remedy is regularization: replace by for . This matrix is positive definite, and the same theorem now optimizes the modified denominator
The resulting direction is proportional to when . It is not a claim that the original singular quotient suddenly became well-defined. In our eight-point example, gives direction , proportional to , instead of . The regularized criterion has changed.
Transform scatter when changing feature coordinates
Let new coordinates be with invertible . The transformed mean difference is , because the translation cancels. Each residual becomes , so the transformed scatter is .
Proposition 19.1.7
(Invariance under invertible coordinate changes)
Let be positive definite, , and with invertible. Then and is positive definite. If is a Fisher direction for the original data, then is a Fisher direction for the transformed data. Moreover and .
Proof
(Transforming the direction and the scores)
For , because is invertible, so is positive definite. Next,
so the maximizers map to the maximizers , that is, . For the scores, : every score shifts by the same constant, and the threshold must shift by it too. Finally and , so the quotient is unchanged. A consistent invertible change of coordinates therefore leaves the unregularized decision boundary unchanged.
Common mistake
(Reusing one regularization strength after rescaling)
The invariance above holds for the unregularized rule only. The matrix generally differs from , so applying the same number after a rescaling is a different penalty. In the eight-point example , and doubles the first diagonal entry. After the unit change the first entry is and the same changes it by only . Decide how features are scaled before choosing a regularization strength.
Exercises with solutions
Quick check
(For the eight-point example, verify the Fisher quotient of , say what happens when is replaced by , and state how to keep the same midpoint classifier.)
Compare the sign of the projected class 1 mean with the direction of the inequality.
Solution · Answer
The projected means differ by 17 and the projected within-class scatter is 272, so the quotient is . Replacing by leaves both squared quantities unchanged. But the projected class-1 mean is now and the class-0 mean is 0. To preserve the original strict rule, predict class 1 when , and send equality to class 0. Keeping a “greater than” comparison would reverse the classifier's labels, despite the identical quotient.
Quick check
(Under the shared-covariance Gaussian model let , , and . Find the equal-cost decision threshold from the log-odds coefficients and explain in which direction it shifts.)
Here .
Solution · Answer
Here and . The rule predicts class 1 when . Compared with equal priors, its threshold is smaller because class 1 is more likely before seeing the features. This formula uses the stated covariance scale and natural logarithm, not an arbitrarily normalized Fisher vector.
Quick check
(Let and . Show that the unregularized maximum of on the positive-denominator domain is . After adding with , which directions are optimal, and what is the maximum?)
Write ; the positive-denominator domain is .
Solution · Answer
For , the positive-denominator domain is . Its quotient is , independently of . Thus all such vectors maximize it, including many nonparallel vectors.
After regularization, the quotient is . Its maximum is , attained exactly when and . Equivalently, the maximizing line is spanned by . The added penalty removes the original null-direction ambiguity in this example.
Quick check
(In the eight-point example change the first feature's unit by , with no translation. Find , and the direction corresponding to . Does the midpoint decision on the same physical point change?)
Use .
Solution · Answer
The transformed mean difference is and scatter is . A consistently scaled direction is . Its score on equals the old score on , so the midpoint threshold stays and the decision is unchanged. Solving the transformed linear system gives , exactly , which represents the same line if the threshold is also divided by 16.