2.4 Fisher discriminants and class scatter

Motivation

A feature can vary substantially without helping us distinguish labelled groups. Suppose two classes have different horizontal centres, while both spread widely in the vertical direction. PCA may retain that vertical variation because it accounts for much of the total variance. A direction designed to distinguish the classes should instead compare separation of their centres with variation inside the classes.

Fisher discriminant analysis makes this comparison explicit. It uses class labels to identify two kinds of variation, then asks for a linear score whose between-class variance is large relative to its within-class variance. The score is still a linear combination of the original features. What changes from PCA is the criterion that selects the combination.

This distinction also changes the geometry. PCA normalizes a direction using its Euclidean length. Fisher directions are naturally normalized by within-class variation. Whitening that variation converts the problem into an ordinary symmetric eigenvalue problem and explains both the maximum ratio and the appropriate orthogonality between several directions.

Definitions

Let $x_1,\ldots,x_n\in\mathbb R^d$ be column vectors with known labels. The index sets $I_1,\ldots,I_K$ partition the observations into $K\ge2$ nonempty classes. Write $n_c=|I_c|$ and $p_c=n_c/n$. A data matrix $X\in\mathbb R^{n\times d}$ has row $i$ equal to $x_i^\top$.

Definition

Class means and empirical class covariance

The class means, overall mean, and class covariance matrices are

$$\mu_c=\frac1{n_c}\sum_{i\in I_c}x_i,\qquad \mu=\frac1n\sum_{i=1}^n x_i=\sum_{c=1}^Kp_c\mu_c,$$$$\Sigma_c=\frac1{n_c}\sum_{i\in I_c} (x_i-\mu_c)(x_i-\mu_c)^\top.$$

Every covariance here uses the empirical denominator, rather than the degrees-of-freedom correction. Class size is therefore part of the weighting: a class containing twice as many observations contributes twice as much when each observation has equal weight. A singleton class has zero empirical covariance because its only observation equals its mean.

Definition

Within-class, between-class, and total variation

Define

$$\begin{aligned} \Sigma_W &=\sum_{c=1}^Kp_c\Sigma_c =\frac1n\sum_{c=1}^K\sum_{i\in I_c} (x_i-\mu_c)(x_i-\mu_c)^\top,\\ \Sigma_B &=\sum_{c=1}^Kp_c(\mu_c-\mu)(\mu_c-\mu)^\top,\\ \Sigma_T &=\frac1n\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^\top. \end{aligned}$$

The corresponding unnormalized scatter matrices are $S_W=n\Sigma_W$ and $S_B=n\Sigma_B$.

For a direction $u\in\mathbb R^d$, the centred score of observation $i$ is $u^\top(x_i-\mu)$. Its class mean is $u^\top(\mu_c-\mu)$. Consequently, $u^\top\Sigma_Wu$ measures the average variance of scores inside their classes, while $u^\top\Sigma_Bu$ measures the weighted variance of the class score means. Multiplying both scatter matrices by the same constant leaves their ratio unchanged.

For comparison with PCA, using $1/(n-1)$ in the overall sample covariance multiplies $\Sigma_T$ by $n/(n-1)$ and preserves its eigenvectors. The decomposition below uses $1/n$ consistently in all three matrices.

Theorem/Proposition

Theorem

Variation decomposes, and class means limit the dimension

The matrices $\Sigma_W$ and $\Sigma_B$ are positive semidefinite, and

$$\Sigma_T=\Sigma_W+\Sigma_B,\qquad \operatorname{rank}(\Sigma_B)\le\min(d,K-1).$$

Thus between-class variation contains at most $K-1$ independent directions, and can contain fewer if the class means occupy a smaller affine subspace.

Assume now that $\Sigma_W$ is positive definite. This means every nonzero feature direction has positive pooled within-class variance. The Fisher ratio is

$$J(u)=\frac{u^\top\Sigma_Bu}{u^\top\Sigma_Wu}, \qquad u\ne0.$$

It is invariant under replacing $u$ by any nonzero multiple of itself. We may therefore impose $u^\top\Sigma_Wu=1$ and maximize the numerator. This normalization fixes a scale for the score without changing its direction.

Theorem

Whitening gives the optimal Fisher directions

For positive-definite $\Sigma_W$, define the symmetric matrix

$$C=\Sigma_W^{-1/2}\Sigma_B\Sigma_W^{-1/2}.$$

Let $v_j$ be orthonormal eigenvectors of $C$, with eigenvalues $\lambda_1\ge\cdots\ge\lambda_d\ge0$. Set

$$u_j=\Sigma_W^{-1/2}v_j.$$

Then

$$\Sigma_Bu_j=\lambda_j\Sigma_Wu_j,\qquad u_i^\top\Sigma_Wu_j=\delta_{ij}.$$

The maximum of $J$ is $\lambda_1$. Successive directions maximize the ratio subject to being $\Sigma_W$-orthogonal to the preceding directions. The number of positive discriminant eigenvalues is $\operatorname{rank}(\Sigma_B)\le\min(d,K-1)$.

For $r$ selected directions, put $U=[u_1\ \cdots\ u_r]$. The reduced representation is $U^\top(x-\mu)$, or $(X-\mathbf1\mu^\top)U$ for the whole data matrix. These are score coordinates. The matrix $UU^\top$ is generally not a Euclidean orthogonal projector, because the columns of $U$ are normalized in the $\Sigma_W$ geometry.

The normalization has a direct interpretation in score space:

$$U^\top\Sigma_WU=I_r,\qquad U^\top\Sigma_BU=\operatorname{diag}(\lambda_1,\ldots,\lambda_r), \qquad U^\top\Sigma_TU=I_r+\operatorname{diag}(\lambda_1,\ldots,\lambda_r).$$

Each selected score has unit pooled within-class variance. Its eigenvalue then measures the between-class variance relative to that common baseline. The cross-covariances between different selected scores vanish in both parts of the decomposition. This is why the normalization is more than a convenient way to shorten a formula: it gives comparable units for different discriminant scores. A larger eigenvalue means a larger standardized contrast, rather than simply a longer coefficient vector.

The sign of a direction remains arbitrary. Changing it reverses scores, class score means, and axis labels together, while preserving every variance and ratio. A repeated eigenvalue permits an orthogonal change of basis inside its whitened eigenspace as well. Thus equivalent calculations can produce different individual coefficient vectors. The appropriate comparison is the normalized discriminant geometry and its eigenvalues, including the selected subspace when the basis inside that subspace is not unique.

Proof sketch or proof idea

For each observation in class $c$, split its displacement as

$$x_i-\mu=(x_i-\mu_c)+(\mu_c-\mu).$$

Expand the outer product and sum within the class. Both cross terms vanish because $\sum_{i\in I_c}(x_i-\mu_c)=0$. Summing the remaining terms over classes gives $\Sigma_T=\Sigma_W+\Sigma_B$. Each quadratic form is a weighted sum of squares, which proves positive semidefiniteness.

Every column of $\Sigma_B$ lies in the span of the centred class means. Those $K$ vectors satisfy the dependence

$$\sum_{c=1}^Kp_c(\mu_c-\mu)=0.$$

Since every $p_c$ is positive, their span has dimension at most $K-1$. Their span can be smaller still: increasing the number of classes does not automatically create new independent directions between their means.

For the optimization, make the change of variable $z=\Sigma_W^{1/2}u$. Then

$$J(u)=\frac{z^\top Cz}{z^\top z}.$$

The ordinary Rayleigh quotient reaches its maximum at a leading eigenvector of $C$. Transforming that vector back gives $u_1$. Orthogonality of the $v_j$ becomes $u_i^\top\Sigma_Wu_j=v_i^\top v_j$, proving the stated metric orthogonality and the successive constrained maximizations.

The same stationary equation follows by differentiating $u^\top\Sigma_Bu-\lambda(u^\top\Sigma_Wu-1)$:

$$2\Sigma_Bu-2\lambda\Sigma_Wu=0.$$

Although one can write $\Sigma_W^{-1}\Sigma_Bu=\lambda u$, that matrix is generally not symmetric in the Euclidean inner product. The whitened matrix $C$ is symmetric. It supplies an orthonormal eigenbasis before transforming back, including a consistent choice inside a repeated-eigenvalue subspace.

When within-class variation can be nonsingular

The amount of data also constrains the within-class covariance. Form a residual matrix $R$ whose row for observation $i$ subtracts the mean of its class $c(i)$. Then

$$R_{i,:}=(x_i-\mu_{c(i)})^\top,\qquad \Sigma_W=\frac1nR^\top R,\qquad \operatorname{rank}(\Sigma_W)\le\min(d,n-K).$$

Within each class, its residual rows sum to zero. The $K$ class-indicator vectors therefore give independent linear constraints on the rows, because their supports are disjoint. This proves the rank bound. If there are fewer than $d$ residual degrees of freedom, positive definiteness is impossible. Having at least $d$ is necessary, but the residuals must also span the feature space for the covariance to be positive definite.

This explains a common high-dimensional situation without introducing an extra distributional model. Many measured features can coexist with relatively few independent within-class residual directions. The singular example below makes that geometry explicit and then specifies exactly which regularized objective is solved.

Two classes give one mean-contrast direction

For two classes, let $\delta=\mu_1-\mu_2$ and suppose $\delta\ne0$. The mean formulas imply

$$\Sigma_B=p_1p_2\delta\delta^\top.$$

Write $a=\Sigma_W^{-1/2}\delta$ and $z=\Sigma_W^{1/2}u$. Cauchy–Schwarz gives

$$J(u)=p_1p_2\frac{(z^\top a)^2}{z^\top z} \le p_1p_2\,a^\top a.$$

Equality holds when $z$ is proportional to $a$, hence

$$u_\star\propto\Sigma_W^{-1}(\mu_1-\mu_2), \qquad J_{\max}=p_1p_2\delta^\top\Sigma_W^{-1}\delta.$$

This is the difference of class means adjusted for within-class variation. A feature with a large mean difference can contribute less after that adjustment if it also varies greatly inside the classes. If the two means coincide, $\Sigma_B=0$ and every defined Fisher ratio is zero.

Worked examples

Worked example

1. Large vertical variance hides horizontal separation from PCA

Take the two classes

$$\begin{aligned} S_1&=\{(-3,-3),(-3,3),(-1,-3),(-1,3)\},\\ S_2&=\{(1,-3),(1,3),(3,-3),(3,3)\}. \end{aligned}$$

The means are $\mu_1=(-2,0)^\top$ and $\mu_2=(2,0)^\top$. The classes have equal size, so $\mu=0$. Within either class, the centred horizontal values are $\pm1$ and the vertical values are $\pm3$, with the cross products cancelling. Therefore

$$\Sigma_W=\begin{bmatrix}1&0\\0&9\end{bmatrix},\qquad \Sigma_B=\begin{bmatrix}4&0\\0&0\end{bmatrix},\qquad \Sigma_T=\begin{bmatrix}5&0\\0&9\end{bmatrix}.$$

The leading PCA direction is vertical because $9>5$. Both classes have the same vertical score values, so those scores do not express their different centres. The leading Fisher direction is horizontal: $u_1=(1,0)^\top$ has $J(u_1)=4$, while the vertical ratio is zero. The horizontal class scores are $\{-3,-1\}$ and $\{1,3\}$. Here the distinction is visible directly in the data, rather than merely in the eigenvalues. Replacing the overall covariance denominator by $n-1$ preserves PCA's vertical direction.

The following NumPy calculation constructs the three covariances from the observations and solves the symmetric whitened problem. Its sign choice makes the horizontal direction point to the right; reversing that sign would reverse every score and preserve the Fisher ratio.

import numpy as np

X = np.array([
    [-3, -3], [-3, 3], [-1, -3], [-1, 3],
    [1, -3], [1, 3], [3, -3], [3, 3]
], dtype=float)
labels = np.repeat([0, 1], 4)
n = len(X)
mu = X.mean(axis=0)
W = np.zeros((2, 2))
B = np.zeros((2, 2))
for label in np.unique(labels):
    group = X[labels == label]
    mean = group.mean(axis=0)
    residual = group - mean
    W += residual.T @ residual / n
    difference = mean - mu
    B += len(group) / n * np.outer(difference, difference)

eigenvalues_W, V = np.linalg.eigh(W)
inverse_root = (V / np.sqrt(eigenvalues_W)) @ V.T
C = inverse_root @ B @ inverse_root
eigenvalues, directions = np.linalg.eigh(C)
u = inverse_root @ directions[:, -1]
if u[0] < 0:
    u = -u
T = (X - mu).T @ (X - mu) / n
assert np.allclose(T, W + B)
assert np.allclose(B @ u, eigenvalues[-1] * (W @ u))
print("W =", W.tolist())
print("B =", B.tolist())
print("direction =", u.tolist())
print("ratio =", float((u @ B @ u) / (u @ W @ u)))
print("W_norm =", float(u @ W @ u))

The output is

W = [[1.0, 0.0], [0.0, 9.0]]
B = [[4.0, 0.0], [0.0, 0.0]]
direction = [1.0, 0.0]
ratio = 4.0
W_norm = 1.0

Worked example

2. Unequal class sizes determine the weights

Now take

$$S_1=\{(-1,0),(1,0)\},\qquad S_2=\{(2,-1),(2,1),(4,-1),(4,1)\}.$$

The class means are $(0,0)^\top$ and $(3,0)^\top$, but the overall mean is $(2,0)^\top$, since $p_1=1/3$ and $p_2=2/3$. The class covariances are $\Sigma_1=\operatorname{diag}(1,0)$ and $\Sigma_2=\operatorname{diag}(1,1)$. Thus

$$\Sigma_W=\operatorname{diag}(1,2/3),\qquad \Sigma_B=\operatorname{diag}(2,0),\qquad \Sigma_T=\operatorname{diag}(3,2/3).$$

For example, the horizontal between-class entry is $\frac13(0-2)^2+\frac23(3-2)^2=2$. The maximum Fisher ratio is $2$, reached horizontally. Giving the two class means equal weight instead would produce mean $3/2$ and between-class entry $9/4$. That is a class-balanced weighting of a different empirical problem. Equal weighting of all six observations requires the factors $1/3$ and $2/3$ throughout the calculation.

Worked example

3. A singular within-class covariance leads to a new regularized objective

Take $S_1=\{(-2,-1),(-2,1)\}$ and $S_2=\{(2,-1),(2,1)\}$. Then

$$\Sigma_W=\operatorname{diag}(0,1),\qquad \Sigma_B=\operatorname{diag}(4,0).$$

There is no horizontal variation within either class, although their horizontal means differ. The ratio at $u=(1,0)^\top$ is undefined because its denominator is zero. For $u=(1,t)^\top$, $t\ne0$, the ratio is $4/t^2$, so it is unbounded as $t$ approaches zero.

Choose $\rho>0$ and define the regularized criterion

$$J_\rho(u)=\frac{u^\top\Sigma_Bu} {u^\top(\Sigma_W+\rho I)u} =\frac{u^\top\Sigma_Bu} {u^\top\Sigma_Wu+\rho\|u\|_2^2}.$$

For $\rho=1/2$, the denominator matrix is $\operatorname{diag}(1/2,3/2)$, which is positive definite. The maximum is $8$, attained horizontally; a normalized direction is $u=(\sqrt2,0)^\top$. This solves the displayed regularized problem. The added term penalizes the Euclidean size of the direction in the denominator, so it changes the criterion rather than merely implementing an inverse for the original one. With $\rho=2$, the maximum becomes $2$.

Interpreting a reduced representation

The informative dimension is controlled by the class centres, not just by the ambient number of features. Two classes supply a single mean contrast even if observations have hundreds of features. With several classes, the first direction captures the strongest standardized contrast; later directions capture additional contrasts subject to the within-class orthogonality constraint. A zero eigenvalue adds no between-class variance.

The labels enter when estimating the means and within-class residuals. Once the directions are fixed, a new observation is mapped by the same linear score formula. The fitted mean and the chosen direction scales belong to that map. Recentring a new observation separately would instead erase its coordinates. Subtracting the single fitted global mean shifts all class scores consistently and preserves all differences between them.

PCA and Fisher analysis therefore answer different questions about the same table. The first preserves overall variation with Euclidean orthogonality. The second preserves labelled mean contrasts relative to within-class variation. The covariance decomposition shows exactly where their criteria overlap and where they differ.

Common mistakes

Common mistake

Using ordinary Euclidean orthogonality for every Fisher direction

The constraint is $u_i^\top\Sigma_Wu_j=0$, rather than generally $u_i^\top u_j=0$. Whitening gives Euclidean-orthogonal vectors $v_j$; transforming them back changes the metric. The symmetric eigenproblem belongs to $C$, not generally to $\Sigma_W^{-1}\Sigma_B$.

Common mistake

Mixing covariance denominators or class weights

The identity $\Sigma_T=\Sigma_W+\Sigma_B$ follows from one consistent empirical weighting. Replacing only the class covariance denominators by $n_c-1$, or averaging unequal classes without their $p_c$ factors, changes the matrices. Write each weighted sum before simplifying it.

Summary

Within-class variation describes dispersion around each labelled class mean, and between-class variation describes weighted differences between those means. Their sum is total variation. A positive-definite within-class covariance turns the Fisher ratio into a symmetric Rayleigh quotient after whitening. The resulting directions are orthogonal in the within-class metric, with at most $K-1$ positive discriminant eigenvalues. For two distinct class means, the direction is their difference adjusted by the inverse within-class covariance. A ridge term defines an explicit modified criterion when that covariance is singular.

Exercises

Checkpoint

1. Let $p_1=1/4$, $p_2=3/4$, $\mu_1=(-1,0)^\top$, $\mu_2=(3,0)^\top$, and $\Sigma_W=\operatorname{diag}(2,5)$. Find $\mu$, $\Sigma_B$, the maximum Fisher ratio, and a normalized maximizing direction.

Use the empirical weights and impose $u^\top\Sigma_Wu=1$.

Checkpoint

2. Take $\Sigma_W=\begin{bmatrix}2&1\\1&1\end{bmatrix}$, $u=(1,0)^\top$, and $v=(1,-2)^\top$. Compute $u^\top v$ and $u^\top\Sigma_Wv$. How can both vectors be normalized in the within-class metric?

Calculate both squared within-class lengths as well.

Checkpoint

3. There are $K=4$ classes in $\mathbb R^5$. What is the largest possible number of positive discriminant eigenvalues when $\Sigma_W$ is positive definite? What if all four class means lie on one affine line?

Count the span of the centred class means.

Checkpoint

4. For the singular covariance in Example 3, use $\rho=2$. Find the maximum of $J_\rho$ and a normalized maximizing direction. What term distinguishes this objective from the original Fisher ratio?

Use the matrix $\Sigma_W+2I=\operatorname{diag}(2,3)$.

Solutions

Solution · 1. Weight the means before normalizing the direction

The overall mean is $\mu=(2,0)^\top$ and $\Sigma_B=\operatorname{diag}(3,0)$. The horizontal ratio is $3/2$ and the vertical numerator is zero, so the maximum is $3/2$. One maximizing direction with unit within-class length is $u=(1/\sqrt2,0)^\top$.

Solution · 2. Orthogonality changes with the metric

The Euclidean inner product is $u^\top v=1$. Since $\Sigma_Wv=(0,-1)^\top$, the within-class inner product is zero. Both squared within-class lengths are $2$: $u^\top\Sigma_Wu=v^\top\Sigma_Wv=2$. Thus $u/\sqrt2$ and $v/\sqrt2$ are within-class orthonormal, while their Euclidean inner product remains $1/2$.

Solution · 3. The mean geometry can sharpen the class-count bound

The general bound is $\min(5,4-1)=3$. On one affine line, all centred class means are multiples of a single vector, so $\operatorname{rank}(\Sigma_B)\le1$ and at most one discriminant eigenvalue is positive. It is one if the class means are not all identical, and zero if they are.

Solution · 4. The ridge changes the denominator and the optimal value

For $u=(a,b)^\top$, $J_2(u)=4a^2/(2a^2+3b^2)\le2$. Equality holds for $b=0$ and $a\ne0$; one normalized direction is $(1/\sqrt2,0)^\top$. The denominator now includes $2\|u\|_2^2$, an added penalty that makes this a different objective from the unregularized ratio.

Continue with SVD and principal components, classical and nonmetric MDS, centred kernels and kernel PCA, or the course reading list.