2.1 SVD and principal components

Motivation

A table with many measured features can contain far fewer independent patterns than its number of columns suggests. Two measurements may move together, or most observations may lie close to a plane inside a much larger feature space. Dimensionality reduction asks for coordinates that describe this structure with fewer numbers. To make that request mathematical, we must specify what the reduced coordinates preserve and how reconstruction error is measured.

Principal component analysis, or PCA, chooses orthogonal directions that retain as much variation as possible. Singular value decomposition, or SVD, gives the coordinates and a precise account of the variation discarded. The same calculation can be understood as fitting a low-dimensional affine subspace to the observations. This connection explains both the centering step and the role of squared singular values.

Unlike regression, PCA does not single out one column as a response variable. All the selected features contribute to the geometry. Consequently, the units used to measure them influence the answer: changing metres to centimetres changes ordinary Euclidean distances unless we also change the preprocessing. Choosing features and their scales is therefore part of specifying the mathematical problem.

Definitions

Let $X\in\mathbb R^{n\times d}$ contain $n\ge2$ observations. Row $i$ is $x_i^T$, where $x_i\in\mathbb R^d$ is a column vector of features. Keep this convention throughout: directions live in feature space $\mathbb R^d$, while a list of scores for all observations lives in $\mathbb R^n$.

Definition

Centering and sample covariance

The mean feature vector and centered matrix are

$$\mu=\frac1nX^T\mathbf1,\qquad H=I_n-\frac1n\mathbf1\mathbf1^T,\qquad A=HX=X-\mathbf1\mu^T.$$

Here $\mathbf1\in\mathbb R^n$ has every entry equal to one. The sample covariance matrix is

$$C=\frac1{n-1}A^TA\in\mathbb R^{d\times d}.$$

The columns of $A$ sum to zero because $\mathbf1^TH=0$. Also $H^T=H$ and $H^2=H$: centering twice changes nothing. Its null space contains the constant vector, so $\operatorname{rank}(A)\le\min(n-1,d)$. This rank bound explains why centering can produce a zero singular value even when the raw table has full row rank. Subtracting the mean removes a location, rather than a genuine direction of variation around that location.

For a unit direction $v\in\mathbb R^d$, the score of observation $i$ is $(x_i-\mu)^Tv$. The score vector is $Av$, whose mean is zero. Its sample variance is therefore

$$\operatorname{Var}_{\mathrm{sample}}(Av) =\frac{\|Av\|_2^2}{n-1}=v^TCv.$$

Definition

Singular values, loadings, and scores

Write a reduced SVD as

$$A=U\Sigma V^T,\qquad p=\min(n,d),$$

where $U\in\mathbb R^{n\times p}$ and $V\in\mathbb R^{d\times p}$ have orthonormal columns, and $\Sigma=\operatorname{diag}(\sigma_1,\ldots,\sigma_p)$ with $\sigma_1\ge\cdots\ge\sigma_p\ge0$. Positive singular values number $r=\operatorname{rank}(A)$. The right singular vectors $v_j$ are PCA loading directions, and $Av_j=\sigma_ju_j$ are their score vectors.

The full SVD instead uses square orthogonal matrices $U$ and $V$, with a rectangular diagonal middle matrix. It supplies additional zero directions when required. For any $k\le r$, both versions give the same leading loading matrix $V_k=[v_1\ \cdots\ v_k]$ and reduced coordinates $Z=AV_k=U_k\Sigma_k$. A score is an observation's coordinate; a loading is a direction used to measure that coordinate. Their different dimensions make them easy to distinguish.

Theorem and proposition

Theorem

Principal directions maximize retained variance

Let $A$ be centered and let $C=A^TA/(n-1)$. Pad the singular-value list with zeros through index $d$ when necessary. The covariance eigenvalues are

$$\lambda_j=\frac{\sigma_j^2}{n-1}.$$

Among unit directions, $v_1$ maximizes score variance. More generally, for $W\in\mathbb R^{d\times k}$ with $W^TW=I_k$, the maximum of $\operatorname{tr}(W^TCW)$ is $\sum_{j=1}^k\lambda_j$, attained by the first $k$ right singular vectors.

For nonzero total variation, the explained-variance fraction of component $j$ and the cumulative fraction of the first $k$ components are

$$\rho_j=\frac{\sigma_j^2}{\sum_{\ell=1}^r\sigma_\ell^2},\qquad R_k=\frac{\sum_{j=1}^k\sigma_j^2}{\sum_{\ell=1}^r\sigma_\ell^2}.$$

The denominator $n-1$ cancels in these fractions. Using a population normalization of $n$ would change covariance eigenvalues, but not principal directions or explained-variance fractions. If every observation is identical, $A=0$: every score is zero and there is no positive total variation from which to form a percentage.

Theorem

Projection error and truncated SVD

For orthonormal $W$, the reconstruction in centered coordinates is $AWW^T$, and

$$\|A-AWW^T\|_F^2 =\|A\|_F^2-\|AW\|_F^2.$$

Thus PCA also minimizes total squared reconstruction error. Its solution is

$$A_k=AV_kV_k^T=U_k\Sigma_kV_k^T,\qquad \|A-A_k\|_F^2=\sum_{j=k+1}^r\sigma_j^2.$$

Moreover, $A_k$ minimizes $\|A-B\|_F$ among all matrices $B$ of rank at most $k$. In the original coordinates, reconstruct with $\widehat X=\mathbf1\mu^T+A_k$.

The squared Frobenius norm is the sum of the squares of all matrix entries. Consequently the error above totals deviations across all observations and all features; it is not a variance per observation or an average per table entry. A chosen rank should be interpreted together with this scale. An eighty-percent retained fraction can be acceptable for one purpose and lose a crucial small-variance feature for another. The optimization states exactly which squared-distance criterion PCA solves.

What the reduced coordinates preserve

The score table has orthogonal columns as well as a zero column mean. In fact,

$$\frac1{n-1}Z^TZ =\frac1{n-1}\Sigma_k^2 =\operatorname{diag}(\lambda_1,\ldots,\lambda_k).$$

Thus different retained scores have zero sample covariance. Their variances appear in decreasing order, so one can read the contribution of each retained coordinate directly from the diagonal. Zero covariance describes the second moments of this score table; it does not, by itself, establish statistical independence of the underlying measurements. PCA constructs orthogonal linear coordinates and optimizes their total variation without specifying a probability model for the observations.

Reconstruction uses the same loading directions in reverse. Given a score row, multiply it by the transpose of the loading matrix to put it back into feature space. The reconstructed centered row lies inside the selected subspace, while its residual is perpendicular to every retained direction. This perpendicularity is a useful computational check: multiplying the residual table by the loading matrix should give zero, up to numerical rounding. The discarded part cannot be recovered from the retained scores alone because the projection maps many different original rows to the same reduced row.

PCA also differs from line fitting with a designated response variable. Its distance to a candidate line is the shortest perpendicular distance in the selected feature metric. Ordinary regression instead measures errors in the response coordinate while treating the explanatory values as given. Even for two features and a one-dimensional fit, these objectives can select different lines. Before applying an algorithm, identify which measurements may vary in the error model and whether a prediction or a geometric representation is the intended output.

Proof sketch or proof idea

Proof

From the spectral theorem to principal components

The symmetric matrix $A^TA$ is positive semidefinite, since $v^TA^TAv=\|Av\|^2\ge0$. Choose an orthonormal eigenbasis, ordered by nonincreasing eigenvalue. For a positive eigenvalue $\sigma_j^2$, set $u_j=Av_j/\sigma_j$. Then $u_i^Tu_j=v_i^TA^TAv_j/(\sigma_i\sigma_j)$, which is one for $i=j$ and zero otherwise. If an eigenvalue is zero, $\|Av_j\|^2=0$, so $Av_j=0$; no division by a zero singular value is needed. The $r$ constructed vectors $u_j$ span the column space of $A$. Complete them to an orthonormal basis of $\mathbb R^n$, giving square $U$, and keep the full feature eigenbasis $V$. Let $\Sigma\in\mathbb R^{n\times d}$ have the positive $\sigma_j$ on its first $r$ diagonal positions and zeros elsewhere. Column by column, $AV=U\Sigma$, hence $A=U\Sigma V^T$. This proves existence even for a rectangular or rank-deficient data matrix.

Expand a unit vector as $v=\sum_j a_jv_j$. Then $v^TCv=\sum_j\lambda_ja_j^2\le\lambda_1$ because $\sum_ja_j^2=1$. For several directions, put $q_j=\|W^Tv_j\|^2$. Orthogonal projection gives $0\le q_j\le1$ and $\sum_jq_j=k$. Hence $\operatorname{tr}(W^TCW)=\sum_j\lambda_jq_j$ is at most the sum of the largest $k$ eigenvalues. Taking their eigenvectors attains the bound.

The projection identity follows observation by observation. Each centered row splits into a part in the chosen subspace and a perpendicular remainder. Pythagoras says their squared lengths add. Summing over rows converts this geometric statement into the Frobenius identity. In SVD coordinates, the discarded rank-one terms are orthogonal under the Frobenius inner product, so each contributes exactly its squared singular value to the error.

To see why allowing an arbitrary rank-$k$ matrix cannot improve the answer, let its rows lie in a subspace $S$ of dimension at most $k$. Projecting each row of $A$ onto $S$ is at least as accurate as choosing that row of the candidate matrix. Among such subspaces, the variance argument has already found the best one. This proves the low-rank statement as well as the geometric PCA statement.

Truncation is also optimal for the spectral norm $\|E\|_2=\max_{\|w\|_2=1}\|Ew\|_2$. For $0\le k<r$,

$$\min_{\operatorname{rank}(B)\le k}\|A-B\|_2 =\|A-A_k\|_2=\sigma_{k+1}.$$

To prove the lower bound, restrict any candidate $B$ of rank at most $k$ to $\operatorname{span}\{v_1,\ldots,v_{k+1}\}$. Its kernel there contains a unit vector $w$, because the domain has dimension $k+1$. Then $Bw=0$ and $\|(A-B)w\|_2=\|Aw\|_2\ge\sigma_{k+1}$. The truncated residual attains this bound: its largest singular value is precisely the first discarded one. The spectral error measures the worst amplification of a unit direction; the squared Frobenius error instead adds squared losses over all discarded directions.

Worked examples

Worked example

1. A complete two-feature PCA calculation

Consider four observations, with rows

$$X=\begin{bmatrix}12&5\\8&5\\10&6\\10&4\end{bmatrix},\qquad \mu=\begin{bmatrix}10\\5\end{bmatrix},\qquad A=\begin{bmatrix}2&0\\-2&0\\0&1\\0&-1\end{bmatrix}.$$

The columns of $A$ are orthogonal, and

$$A^TA=\begin{bmatrix}8&0\\0&2\end{bmatrix},\qquad C=\begin{bmatrix}8/3&0\\0&2/3\end{bmatrix}.$$

Choose $v_1=(1,0)^T$, $v_2=(0,1)^T$. Then $\sigma_1=2\sqrt2$, $\sigma_2=\sqrt2$, and

$$U=\begin{bmatrix} 1/\sqrt2&0\\-1/\sqrt2&0\\0&1/\sqrt2\\0&-1/\sqrt2 \end{bmatrix},\qquad V=I_2.$$

Multiplying $U\operatorname{diag}(2\sqrt2,\sqrt2)V^T$ reproduces $A$. The first score vector is $(2,-2,0,0)^T$, with sample variance $8/3$. It retains $8/(8+2)=4/5$ of the total variation. Keeping only this coordinate and restoring the mean gives

$$\widehat X= \begin{bmatrix}12&5\\8&5\\10&5\\10&5\end{bmatrix}.$$

The last two observations each have an error of magnitude one. Total squared error is $2$, agreeing with the discarded value $\sigma_2^2$. Averaged over the eight entries, the squared error is $1/4$, so the entrywise root mean square error is $1/2$.

This example separates four quantities often called the size of a component: the singular value, its squared value, the sample variance, and the explained fraction. They describe the same direction but answer different questions. The singular value is a length in score space; division of its square by $n-1$ gives a sample variance. Ratios require the squares because orthogonal energy, rather than unsquared length, is additive.

Worked example

2. Centering changes the fitted geometry

For the same raw observations,

$$X^TX=A^TA+n\mu\mu^T =\begin{bmatrix}408&200\\200&102\end{bmatrix}.$$

The off-diagonal entry is now $200$, so $(1,0)^T$ is not an eigenvector of $X^TX$. An SVD of $X$ would select a different leading direction, influenced by the data's location relative to the origin. Projecting the raw rows onto the horizontal axis through the origin alone would discard their entire second coordinate, producing squared error $25+25+36+16=102$.

Centered PCA instead fits the horizontal line through $(10,5)$ and has error $2$. These are different fitting problems. The best raw rank-one direction need not be horizontal; the comparison explains why the raw horizontal projection is not the centered reconstruction.

For a new observation $(11,7)$, subtract the training mean to obtain $(1,2)$. Its first score is $1$ and its one-component reconstruction is $(11,5)$. Use the same mean and loading directions when transforming additional observations, so their scores share one coordinate system.

The affine viewpoint is useful here. The retained centered subspace passes through zero, but adding the mean translates it back to the part of feature space occupied by the data. Centering therefore allows the dimensions to describe variation around the observations, rather than spending a direction to describe their displacement from zero. A reduced score table alone cannot restore the original location: the mean vector is also needed for decoding.

Worked example

3. Changing feature units and identifying a tie

Multiply the second feature by three. After centering, the new matrix has rows $(2,0)$, $(-2,0)$, $(0,3)$, $(0,-3)$, so its scatter matrix is $\operatorname{diag}(8,18)$. Its first loading is now $(0,1)^T$, the retained fraction is $18/26=9/13$, and the rank-one squared error is $8$.

Alternatively, standardize each original centered column by its sample standard deviation. Those standard deviations are $\sqrt{8/3}$ and $\sqrt{2/3}$. The standardized scatter matrix is $3I_2$, so its covariance matrix is $I_2$. Every unit direction has score variance one. There is no preferred single first component; any orthonormal pair spans the same two-dimensional space.

Standardization selects a different metric by giving features comparable scales. It may be useful when units are arbitrary, but it also removes meaning from an intentional difference in feature variability. State whether the objective measures error in original units or standardized units before interpreting the retained fraction. A zero-variance feature has no positive standard deviation to divide by; it contributes no centered variation.

Power iteration and the pseudoinverse

From a real eigenbasis to power iteration

The eigenvalue definitions, real diagonalization, and diagonalization tests provide the prerequisites. Consider a real square matrix $B=VDV^{-1}$ with a real eigenbasis, taking each column $v_j$ of $V$ to have unit length. The eigenvectors need not be orthogonal. A sufficient dominant-modulus condition is

$$|\lambda_1|>\max_{j>1}|\lambda_j|,\qquad c=V^{-1}q_0,\qquad c_1\ne0.$$

Here $q_0$ is a unit initial vector. Normalize after each multiplication and expand its unnormalized powers in the fixed eigenbasis:

$$q_{k+1}=\frac{Bq_k}{\|Bq_k\|_2},\qquad B^kq_0=c_1\lambda_1^k \left(v_1+\sum_{j>1}\frac{c_j}{c_1} \left(\frac{\lambda_j}{\lambda_1}\right)^k v_j\right).$$

The remaining coefficients decay geometrically relative to the first, so the eigenvector line is approached. The nonzero leading coefficient also ensures that the normalization denominator cannot vanish. The ratios concern the eigenbasis coordinates; bounds in Euclidean norm additionally depend on the geometry of that basis. The sharper orthogonal-coordinate estimate below is available for a symmetric scatter matrix.

The coefficient condition means solving $Vc=q_0$, rather than taking a dot product with $v_1$. For a nonorthogonal eigenbasis, $c_1$ can be nonzero even when $v_1^Tq_0=0$. Conversely, a nonzero dot product does not ensure a nonzero first eigenbasis coordinate. If that coordinate is absent, multiplication does not create it. The strict modulus gap excludes a second mode of equal asymptotic size, which could otherwise persist alongside the first.

When $\lambda_1<0$, normalized vectors can alternate signs while approaching the same line. The Rayleigh quotient $\rho(q)=q^TBq/(q^Tq)$ still tends to $\lambda_1$: it is continuous near either unit representative of that line and has the same value at opposite signs. For nonsymmetric $B$, this limit does not supply the extremal-variance interpretation available for a symmetric matrix. In particular, intermediate quotient values need not lie between the smallest and largest real eigenvalues.

Worked example

A negative dominant eigenvalue with a nonorthogonal basis

Take

$$B=\begin{bmatrix}-3&4\\0&1\end{bmatrix},\qquad V=\begin{bmatrix}1&1\\0&1\end{bmatrix},\qquad D=\operatorname{diag}(-3,1).$$

These displayed basis columns are unnormalized eigenvectors. With $q_0=(0,1)^T$, their coordinates are $(-1,1)^T$, so the dominant coefficient is present. Normalizing the basis columns rescales these coefficients without changing which ones are zero. The raw iterates are $(4,1)^T$ and $(-8,1)^T$; $q_2=(-8,1)^T/\sqrt{65}$ has $\rho(q_2)=-223/65<-3$. The dominant line is horizontal, and the signs alternate. Starting instead along $(1,1)^T$ stays in the eigenvalue-one direction despite its nonzero dot product with the horizontal eigenvector.

For a unit right eigenvector, subtracting $\lambda_1v_1v_1^T$ still gives eigenvalue zero along $v_1$, but usually changes the other original eigenvectors. To preserve their eigenpairs, take a left eigenvector with $w_1^TB=\lambda_1w_1^T$ and $w_1^Tv_1=1$, and subtract $\lambda_1v_1w_1^T$. In the example, $v_1=(1,0)^T$ and $w_1=(1,-1)^T$ give

$$B+3v_1v_1^T=\begin{bmatrix}0&4\\0&1\end{bmatrix},\qquad B+3v_1w_1^T=\begin{bmatrix}0&1\\0&1\end{bmatrix}.$$

The first sends the original eigenvector $(1,1)^T$ to $(4,1)^T$; the paired update preserves it. For a symmetric matrix, left and right unit eigenvectors can coincide, recovering the familiar orthogonal deflation formula.

Approximating a dominant PCA direction

When only the first principal direction is needed, power iteration can avoid a full decomposition. Let $S=A^TA$ have eigenvalues $\lambda_1>\lambda_2\ge\cdots\ge\lambda_d\ge0$ and orthonormal eigenvectors $v_j$. Start with a unit vector $q_0=\sum_jc_jv_j$ whose first coefficient $c_1$ is nonzero, and repeat

$$q_{k+1}=\frac{Sq_k}{\|Sq_k\|_2}.$$

Each step amplifies the first component relative to the others. Before normalization, the iterate is proportional to $S^kq_0$; after factoring out $\lambda_1^k$, every other eigenvector's coefficient contains $(\lambda_j/\lambda_1)^k$. The ratio of the perpendicular part to the leading part is at most

$$\left(\frac{\lambda_2}{\lambda_1}\right)^k \frac{\sqrt{\sum_{j=2}^d c_j^2}}{|c_1|}.$$

Thus the direction converges geometrically to $v_1$ up to sign. Normalizing keeps vector lengths manageable without changing this ratio. A small spectral gap gives slower convergence; an initial vector perpendicular to the first direction never gains that component. For a unit iterate, $q_k^TSq_k$ estimates the leading eigenvalue. Once an exact leading pair is known, $S-\lambda_1v_1v_1^T$ removes that mode. With a large data table, evaluate $Sq=A^T(Aq)$ without explicitly forming the scatter matrix.

Worked example

4. Following the dominant scatter direction

Take $S=\operatorname{diag}(4,1)$ and $q_0=(1,1)^T/\sqrt2$. Two iterations give

$$q_1=\frac{(4,1)^T}{\sqrt{17}},\qquad q_2=\frac{(16,1)^T}{\sqrt{257}}.$$

The second coordinate divided by the first is successively $1$, $1/4$, and $1/16$. These ratios exhibit the spectral factor $\lambda_2/\lambda_1$. The iterates approach the first PCA loading $(1,0)^T$.

Choosing a minimum-norm least-squares coefficient

SVD also resolves coefficient ambiguity in a fit. For a real matrix $M\in\mathbb R^{m\times p}$, retain its positive singular values in $M=U_r\Sigma_rV_r^T$. Define

$$M^+=V_r\Sigma_r^{-1}U_r^T,\qquad z_*=M^+y.$$

Direct SVD multiplication verifies the four Penrose identities:

$$MM^+M=M,\qquad M^+MM^+=M^+,\qquad (MM^+)^T=MM^+,\qquad(M^+M)^T=M^+M.$$

In particular, $MM^+=U_rU_r^T$ projects orthogonally onto the column space, while $M^+M=V_rV_r^T$ projects onto the row space. Both are symmetric and idempotent. The first is $I_m$ for full row rank; the second is $I_p$ for full column rank. For a square invertible matrix these become the usual two-sided inverse relations, and $M^+=M^{-1}$. For a singular fit, the projections identify exactly which response and coefficient directions the data determine.

For any response $y$, this is the unique minimum-norm least-squares coefficient. Indeed, $Mz_*=U_rU_r^Ty$ is the orthogonal projection of $y$ onto the column space of $M$. It attains the smallest residual. Every other least-squares coefficient is $z_*+h$ with $h\in\ker M$. Since $z_*$ lies in the span of $V_r$, it is perpendicular to $\ker M$, giving $\|z_*+h\|_2^2=\|z_*\|_2^2+\|h\|_2^2$. Equality of norms requires $h=0$.

When the equations are consistent, the residual vanishes and this chooses the shortest exact solution. Otherwise it chooses the shortest coefficient among all equally accurate fits. Taking reciprocals only of positive singular values distinguishes this construction from trying to invert a singular matrix. The same SVD supplies both data-reduction directions and a precise rule for selecting coefficients along directions the observations determine.

In left singular coordinates, the coefficient of $v_j$ is $(u_j^Ty)/\sigma_j$. A portion of $y$ perpendicular to the column space cannot be fitted and remains in the residual; zero right singular directions create the freedom to add null-space vectors. The minimum-norm choice sets those free coordinates to zero. This coordinate description explains both the fit and the coefficient selection through one division per positive singular value. Keeping only some positive singular directions instead solves a restricted fitting problem in their selected span, rather than the same unrestricted minimum-norm least-squares problem.

Worked example

5. Resolving an underdetermined fit

Let $M=[1\ \ 2]$ and $y=5$. Then $M^+=(1,2)^T/5$, so $z_*=(1,2)^T$. All exact solutions are $z=(5-2t,t)^T$, and

$$\|z\|_2^2=(5-2t)^2+t^2=5(t-2)^2+5.$$

The minimum occurs at $t=2$, confirming $z_*$ and its squared norm $5$. The difference from this solution is $(t-2)(-2,1)^T$, a null-space direction perpendicular to $(1,2)^T$.

Computing scores and reconstructions

The following calculation implements the first example. Vt stores loading directions as rows, because the numerical SVD returns the transpose of $V$.

import numpy as np

X = np.array([[12., 5.], [8., 5.], [10., 6.], [10., 4.]])
mean = X.mean(axis=0)
A = X - mean
U, s, Vt = np.linalg.svd(A, full_matrices=False)
k = 1
scores = A @ Vt[:k].T
reconstruction = scores @ Vt[:k] + mean
fractions = s**2 / np.sum(s**2)
squared_error = np.sum((X - reconstruction)**2)
print(fractions)      # [0.8 0.2]
print(squared_error)  # 2.0, up to rounding

A numerical routine can flip the sign of a loading and the corresponding score column. Their product, the reconstruction, remains unchanged. Compare subspaces, explained variances, and reconstructed matrices when checking two implementations. If singular values tie across the chosen cutoff, even the optimal retained subspace can be nonunique. With a strict gap at that cutoff, the subspace is determined even though the signs of its basis vectors are not.

Common mistakes

Common mistake

Using singular values directly as variance percentages

For the first example, the unsquared ratio $2\sqrt2/(2\sqrt2+\sqrt2)=2/3$ is not the retained variance fraction. The correct value is $8/(8+2)=4/5$. Also distinguish $V_k$, whose columns describe feature directions, from $AV_k$, whose rows describe observations. Confusing the two usually creates a dimension mismatch before it creates a plausible interpretation.

Summary

Center the columns, select a metric, and compute the SVD of the centered matrix. Right singular vectors supply loading directions; multiplying by them supplies scores. Covariance eigenvalues are squared singular values divided by $n-1$. The first components maximize retained variance and minimize orthogonal squared reconstruction error. Restore the mean when returning to the original feature coordinates.

Exercises

Checkpoint

1. For the four-observation example, retain one component. Give its explained fraction, total squared reconstruction error, and the reconstruction of the new point (11,7).

Separate centering, projection, and restoring the mean.

Checkpoint

2. A centered matrix has positive singular values 6, 3, and 1. What is the smallest retained rank giving at least 90% explained variance? What squared reconstruction error remains?

Form cumulative sums of squared singular values.

Checkpoint

3. A covariance matrix has eigenvalues 5, 5, and 0. Is a first loading uniquely determined? Is its leading two-dimensional subspace uniquely determined?

Compare a tie inside a retained subspace with a tie across the truncation cutoff.

Solutions

Solution · 1. Projection and restoring the mean

The explained fraction is $4/5$, and squared error is $2$. The new point has centered coordinates $(1,2)$ and first score $1$. Its centered reconstruction is $(1,0)$; adding $(10,5)$ gives $(11,5)$.

Solution · 2. Choosing a rank from the squared spectrum

Total squared singular value is $36+9+1=46$. One component retains $36/46$, below $0.9$. Two retain $45/46$, above $0.9$, so the smallest rank is two. The discarded squared singular value is $1$, which is the remaining total squared reconstruction error.

Solution · 3. A nonunique basis and a determined subspace

Every unit vector in the eigenvalue-five eigenspace maximizes the variance, so a first loading is not unique, even after accounting for sign. The full two-dimensional eigenspace is unique because its eigenvalue is separated from zero. Any orthonormal basis of this plane retains total variance $10$; changing that basis rotates the scores without changing their reconstruction.

Related notes

• Least squares and residuals
• Gradient descent on quadratics
• Course reading