Symmetry gives orthogonal eigenvectors

What an orthogonal diagonalization means

A symmetric matrix may contain off-diagonal entries because the current coordinate axes are not its natural axes. An orthogonal change of coordinates separates its quadratic form into independent squared coordinates while preserving lengths and angles. Throughout this unit the scalar field is real, $n\ge1$, and $A=A^T\in\mathbb R^{n\times n}$. No distinct-eigenvalue assumption is imposed.

Definition

Orthogonality, eigenvectors and spectrum

For $u,v\in\mathbb R^n$, orthogonality means $u^Tv=0$. A list $q_1,\ldots,q_n$ is an orthonormal basis if $q_i^Tq_j=\delta_{ij}$, where $\delta_{ij}$ is one when $i=j$ and zero otherwise. A square real matrix $Q$ is orthogonal when $Q^TQ=I_n$. It is then invertible with inverse $Q^T$, so also $QQ^T=I_n$.

An eigenvector is a nonzero vector $q$ satisfying $Aq=\lambda q$. Its eigenvalue is $\lambda$, and the spectrum is the set of eigenvalues. A diagonal matrix $D=\operatorname{diag}(\lambda_1,\ldots,\lambda_n)$ may list an eigenvalue repeatedly. If the columns of $Q$ are an orthonormal eigenbasis, then $AQ=QD$ and consequently

$$Q^TAQ=D,\qquad A=QDQ^T.$$

Conversely, these identities with orthogonal $Q$ say exactly that its columns form such an eigenbasis. Orthogonality concerns the columns and their lengths; it does not require $\det Q=1$.

Writing $x=Qy$ gives $\|x\|^2=\|y\|^2$ and $x^TAx=\sum_{i=1}^n\lambda_i y_i^2$. Thus the spectral theorem describes both the linear map and its quadratic form. The proof below uses compactness of the unit sphere, the regular norm constraint, Gram–Schmidt and induction; it does not assume the desired eigenbasis in advance. Distinct eigenvalues force orthogonal eigenspaces. Within a repeated eigenspace, one must still choose an orthonormal basis.

4.3 Symmetry gives orthogonal eigenvectors

For real symmetric $A$ and real vectors $u,v$, if $Au=\lambda u$, $Av=\mu v$, then

$$\lambda u^Tv=(Au)^Tv=u^TA^Tv=u^TAv=\mu u^Tv.$$

Distinct eigenvalues therefore have orthogonal eigenspaces. The stronger spectral theorem states that every real symmetric $n\times n$ matrix, where $n\ge1$, has an orthonormal eigenbasis, hence $A=QDQ^T$ with $Q^TQ=I$ and real diagonal $D$.

Here is a proof using earlier chapters. The continuous function $x^TAx$ attains a maximum on the unit sphere, which is closed and bounded. At a maximizing unit vector $v$, the regular constraint $x^Tx=1$ has nonzero gradient $2v$. The multiplier condition gives $2Av=2\lambda v$, so $v$ is a real unit eigenvector. Its perpendicular subspace is invariant: if $v^Tw=0$, then $v^TAw=(Av)^Tw=\lambda v^Tw=0$. Restrict $A$ to that $(n-1)$-dimensional subspace in an orthonormal basis; the restriction remains symmetric. Induction, beginning with dimension one, supplies an orthonormal eigenbasis there. Together with $v$, it proves the theorem. An orthonormal basis for the perpendicular subspace is constructed by the Gram–Schmidt method.

Real inner products, their norms and the needed inequalities

An inner product on a real vector space is a map $\langle u,v\rangle\in\mathbb R$ that is symmetric, linear in each argument, and positive definite: $\langle v,v\rangle>0$ for $v\ne0$. Linearity gives $\langle0,v\rangle=0$. Define $\|v\|=\sqrt{\langle v,v\rangle}$. It is zero exactly at zero and satisfies $\|cv\|=|c|\|v\|$. The standard example on $\mathbb R^m$ is $\langle u,v\rangle=\sum_{i=1}^mu_iv_i$. Vectors are orthogonal when their inner product is zero, and a list is orthonormal when distinct members are orthogonal and each member has norm one.

If $v\ne0$, insert $t=\langle u,v\rangle/\|v\|^2$ into the nonnegative square $\|u-tv\|^2$. Expanding by bilinearity gives $\|u\|^2-\langle u,v\rangle^2/\|v\|^2\ge0$, hence $|\langle u,v\rangle|\le\|u\|\|v\|$. If $v=0$ the same inequality holds directly. Equality for $v\ne0$ means $\|u-tv\|=0$, equivalently $u=tv$; the zero-vector cases also give equality. Finally expand $\|u+v\|^2$ and apply the bound to its cross term to obtain $\|u+v\|^2\le(\|u\|+\|v\|)^2$. Both sides to be square-rooted are nonnegative, so the triangle inequality follows. Thus this is a norm, and normalization of any nonzero vector divides by a positive real number. No complex bilinear convention is being used.

Finite basis existence and extension, without a counting assumption

Let $V$ be a real vector space supplied with a finite spanning list $t_1,\ldots,t_N$. First prove that every independent list $u_1,\ldots,u_k$ has length at most $N$. After $j$ insertions, maintain a spanning list consisting of $u_1,\ldots,u_j$ and $N-j$ surviving old vectors. Expand $u_{j+1}$ in this list. Some coefficient on a surviving old vector is nonzero; otherwise $u_{j+1}$ would be in the span of the earlier $u_i$, contradicting independence. Solve for that old vector and replace it by $u_{j+1}$. The new list spans the same space and keeps every earlier insertion. If $k>N$, after $N$ steps the first $N$ independent vectors already span $V$, contradicting independence of $u_{N+1}$. This proves the bound, including $N=0$.

Every subspace $W$ has a finite basis. Start with the empty independent list. If its current span is not $W$, choose one vector of $W$ outside that span. A relation with nonzero coefficient on the new vector would express it in the old span, so the enlarged list remains independent. The proved bound permits at most $N$ additions; therefore the process stops with an independent spanning list for $W$. If $W=\{0\}$, it stops immediately with the empty basis. Starting instead with any given independent list in $W$ proves that list extends to a basis of $W$. Only a bounded finite selection is used. Finally, if two bases have sizes $p,q$, use each as the finite spanning list in the bound to obtain $p\le q$ and $q\le p$. Thus basis size, called dimension, is well-defined. This supplies the existence, extension and dimension facts used in rank–nullity.

Orthonormal bases and extension while retaining prescribed vectors

Suppose in addition that $V$ has a real positive-definite inner product. On an independent list $v_1,\ldots,v_s$, define successively

$$w_j=v_j-\sum_{i<j}\langle v_j,q_i\rangle q_i,\qquad q_j=w_j/\|w_j\|.$$

Inductively the previous $q_i$ are orthonormal. For $l<j$, taking the inner product with $q_l$ gives $\langle w_j,q_l\rangle=\langle v_j,q_l\rangle-\langle v_j,q_l\rangle=0$. If $w_j=0$, then $v_j$ lies in the previous span, a contradiction. Division by its positive norm is therefore legal. The displayed formula and its rearrangement show that the first $j$ old and new vectors span the same space, completing the induction. Apply this to a basis to obtain an orthonormal basis, also for a subspace. To extend a prescribed orthonormal list, first extend it to an ordinary basis by the proved theorem and then apply this construction in that order. On the prescribed first vectors no component is subtracted and the norm is one, so every prescribed vector is retained exactly. Empty lists and the zero subspace require no division. This proves the orthonormal extension used in SVD and projection existence.

Fix an integer $n\ge1$ and the Euclidean norm on $\mathbb R^n$. A set is bounded if it lies in some finite-radius ball, closed if its complement is open, and compact if every cover by open subsets of $\mathbb R^n$ has a finite subcover. We use real completeness, including the nested closed interval property.

Heine–Borel in finite-dimensional Euclidean space

Theorem. A set $K\subseteq\mathbb R^n$ is compact if and only if it is closed and bounded. The empty set satisfies all three properties.

Compact implies bounded. The open balls $B(0,m)$, $m\ge1$ an integer, cover $K$. A finite subcover puts $K$ inside the largest of these balls.

Compact implies closed. Suppose $K\ne\varnothing$ and $y\notin K$. The balls $B(x,\|x-y\|/2)$, $x\in K$, cover $K$. Choose a finite subcover with centers $x_1,\ldots,x_r$ and set $\delta=\min_i\|x_i-y\|/2>0$. If $z$ lies in the ball centered at $x_i$, the triangle inequality gives $\|z-y\|>\|x_i-y\|/2\ge\delta$. Hence $B(y,\delta)$ misses $K$, so the complement is open.

Closed and bounded implies compact. Let nonempty $K$ be closed and bounded. Enclose it in a closed cube $Q_0=[-M,M]^n$ with $M>0$. Suppose an open cover of $K$ has no finite subcover. Divide each coordinate interval into two closed halves. The resulting $2^n$ closed subcubes cover $Q_0$. At least one subcube has intersection with $K$ that is not finitely coverable by the given family; otherwise the finite union of the finite subcovers would cover $K$. Select such a subcube and repeat. This produces nested closed cubes $Q_j$ with side length $2M/2^j$ and $K\cap Q_j$ not finitely coverable, hence nonempty.

For each coordinate, the nested interval property gives a common coordinate value. Together these give $p\in\bigcap_jQ_j$. Every pair of points in $Q_j$ has distance at most $2M\sqrt n/2^j$, so the intersection contains only $p$. Choose $x_j\in K\cap Q_j$. Then $\|x_j-p\|\le2M\sqrt n/2^j\to0$. A closed set contains such a limit: if $p$ were in its open complement, a ball around $p$ would eventually contain $x_j$ outside $K$, a contradiction. Thus $p\in K$.

One member $O$ of the cover contains $p$, and openness gives $B(p,\varepsilon)\subseteq O$ for some $\varepsilon>0$. For sufficiently large $j$, every point of $Q_j$ is within $\varepsilon$ of $p$, so $O$ alone covers $K\cap Q_j$. This contradicts the selection of $Q_j$. Therefore every open cover has a finite subcover. The number $2^n$ is finite; replacing it by infinitely many subpieces would invalidate this argument.

Continuous images and extreme values

Theorem. If $K\subseteq\mathbb R^n$ is compact and $F:K\to\mathbb R^m$ is continuous relative to $K$, then $F(K)$ is compact. In particular, a continuous $f:K\to\mathbb R$ on nonempty compact $K$ is bounded and attains its maximum and minimum.

Proof. An open cover $(O_i)$ of $F(K)$ pulls back to relatively open sets $F^{-1}(O_i)=K\cap U_i$, with $U_i$ open in $\mathbb R^n$. These $U_i$ cover $K$, so finitely many suffice. Their corresponding $O_i$ cover $F(K)$, proving compactness. For scalar $f$, the image is nonempty, closed and bounded by the theorem already proved in dimension one. Let $a=\sup f(K)$. For each $j\ge1$, the definition of supremum supplies $y_j\in f(K)$ with $a-1/j<y_j\le a$. Then $y_j\to a$, and closedness places $a$ in the image. Thus the maximum is attained. Applying the same argument to $-f$ gives the minimum. This proves existence, not a formula for the maximizing input.

A direct algebraic proof of the real symmetric spectral theorem

Let $n\ge1$ be a finite integer and $A=A^T\in\mathbb R^{n\times n}$. Use the standard real inner product. A list spans a space when every vector is a linear combination of its members; it is independent when a zero linear combination forces every coefficient to be zero. A basis has both properties. A subspace contains zero and is closed under addition and real scalar multiplication. The coordinate vectors $e_1,\ldots,e_n$ form a basis of $\mathbb R^n$, as is seen by reading each coordinate. The preceding finite-basis and orthonormal-extension proofs therefore apply to this space and every one of its subspaces.

The triangle inequality in the preceding inner-product proof, applied in both directions, gives $|\|x\|-\|y\||\le\|x-y\|$. Thus the norm is continuous. The unit sphere is nonempty because it contains $e_1$ when $n\ge1$, and it is bounded because each of its points has norm one. It is closed: if $\|x\|\ne1$, the ball of radius $|\|x\|-1|/2$ about $x$ misses the sphere, by the reverse triangle inequality. The proved finite-dimensional Heine–Borel theorem therefore makes the sphere compact. Since $x^TAx$ is a continuous polynomial, the proved extreme-value theorem supplies a maximum there. Choose a maximizing unit vector $u$ and put $\lambda=u^TAu$. For any $w$ with $u^Tw=0$, set $b=w^TAu$, $c=w^TAw$ and $d=w^Tw\ge0$. For every real $s$, the vector $(u+sw)/\sqrt{1+s^2d}$ is a unit vector. Maximality and symmetry give

$$\frac{\lambda+2sb+s^2c}{1+s^2d}\le\lambda, \qquad 2sb+s^2(c-\lambda d)\le0.$$

If $b\ne0$, put $k=c-\lambda d$ and choose $s=b/(|k|+1)$. Then

$$2sb+s^2k=\frac{b^2[2(|k|+1)+k]}{(|k|+1)^2}>0,$$

because $2(|k|+1)+k\ge |k|+2>0$. This contradiction proves $w^TAu=0$ for every $w\perp u$. In particular, take $w=Au-\lambda u$. It is perpendicular to $u$, and $w^TAu=w^T(w+\lambda u)=\|w\|^2$. Thus $w=0$ and $Au=\lambda u$. This establishes the first real eigenvector directly from maximality.

For $z\perp u$, symmetry gives $u^TAz=(Au)^Tz=\lambda u^Tz=0$, so $W=u^\perp$ is invariant. Extend $u$ to an orthonormal basis $u,q_2,\ldots,q_n$ by the preceding proved extension construction. Every vector has a unique expansion in this basis, and its coefficient on $u$ is its inner product with $u$. Hence $q_2,\ldots,q_n$ form an orthonormal basis of $W$, including the empty basis when $n=1$.

Induct on $n$. Dimension one is immediate for a real scalar matrix. If $n>1$, the matrix of the restriction to $W$ has entries $B_{ij}=q_{i+1}^TAq_{j+1}$ for $1\le i,j\le n-1$. These entries are real and $B_{ij}=B_{ji}$. The induction hypothesis gives an orthonormal eigenbasis of this real symmetric $(n-1)\times(n-1)$ matrix. Replacing each coordinate vector $(r_1,\ldots,r_{n-1})$ of that basis by $\sum_{j=1}^{n-1}r_jq_{j+1}$ preserves inner products, by bilinearity and $q_i^Tq_j=\delta_{ij}$. The definition of $B$ and invariance of $W$ show that these are eigenvectors of $A$ in $W$. Together with $u$ they form an orthonormal eigenbasis of $\mathbb R^n$.

Writing this basis as $p_1,\ldots,p_n$, every $x$ satisfies $x=\sum_i(p_i^Tx)p_i$ and $Ax=\sum_i\lambda_i(p_i^Tx)p_i$. Coordinatewise this is $A=QDQ^T$, where $Q$ has columns $p_i$ and $D=\operatorname{diag}(\lambda_i)$. Orthonormality gives $Q^TQ=I$, and the expansion of $x$ gives $QQ^T=I$. No distinct-eigenvalue assumption is used. The argument uses real completeness through compactness, finite-dimensional basis extension and inner-product algebra; it uses no multiplier or implicit-function theorem.

A complete two-by-two diagonalization

Worked example

Finding and checking every factor

Take $A=\begin{pmatrix}2&1\\1&2\end{pmatrix}$. Its characteristic equation is $(2-\lambda)^2-1=(\lambda-3)(\lambda-1)=0$. For $\lambda=3$, solving $(A-3I)q=0$ gives the line spanned by $(1,1)^T$; for $\lambda=1$, it gives the line spanned by $(1,-1)^T$. Normalize these vectors:

$$q_1=\frac1{\sqrt2}\binom11,\qquad q_2=\frac1{\sqrt2}\binom1{-1},\qquad Q=\frac1{\sqrt2}\begin{pmatrix}1&1\\1&-1\end{pmatrix},\qquad D=\begin{pmatrix}3&0\\0&1\end{pmatrix}.$$

Both columns have length one and their dot product is zero. Explicitly,

$$Q^TQ=\frac12\begin{pmatrix}2&0\\0&2\end{pmatrix}=I_2,\qquad AQ=\frac1{\sqrt2}\begin{pmatrix}3&1\\3&-1\end{pmatrix}=QD.$$

Finally check the reconstruction, rather than stopping at the eigenvalues:

$$QDQ^T=\frac12\begin{pmatrix}3&1\\3&-1\end{pmatrix}\begin{pmatrix}1&1\\1&-1\end{pmatrix}=\frac12\begin{pmatrix}4&2\\2&4\end{pmatrix}=A.$$

Here $\det Q=-1$, which is permitted. In coordinates $y=Q^Tx$, the quadratic form is $3y_1^2+y_2^2$; the off-diagonal term $2x_1x_2$ has been absorbed into the choice of axes. Changing a column's sign or reordering both the columns and their corresponding diagonal entries gives another valid factorization.

Exercise S01: Rayleigh bounds and equality

For real symmetric $A$ with $n\ge1$, let $\lambda_{\min}$ and $\lambda_{\max}$ be its smallest and largest eigenvalues. Prove that for every $x\ne0$,

$$\lambda_{\min}\le\frac{x^TAx}{x^Tx}\le\lambda_{\max},$$

and characterize both equality cases, allowing repeated eigenvalues.

Solution · S01: expand in an orthonormal eigenbasis

Write $x=\sum_i y_iq_i$ in the eigenbasis just proved to exist. Then $x^Tx=\sum_i y_i^2>0$ and $x^TAx=\sum_i\lambda_i y_i^2$. The quotient is a weighted average of the eigenvalues with nonnegative weights $y_i^2/\sum_jy_j^2$ summing to one; this proves both inequalities. For the upper equality,

$$\lambda_{\max}x^Tx-x^TAx=\sum_i(\lambda_{\max}-\lambda_i)y_i^2=0.$$

Each summand is nonnegative, so $y_i=0$ whenever $\lambda_i<\lambda_{\max}$. Thus equality holds exactly for nonzero vectors in the entire $\lambda_{\max}$ eigenspace, not just a chosen eigenvector. Replacing the differences by $\lambda_i-\lambda_{\min}$ proves the lower equality characterization. If all eigenvalues coincide, both equalities hold for every nonzero vector. The quotient is undefined at $x=0$, though the corresponding unnormalized inequalities hold there with equality.

Exercise S02: read definiteness from the spectrum

A real symmetric matrix is positive semidefinite if $x^TAx\ge0$ for every real $x$, and positive definite if $x^TAx>0$ for every $x\ne0$. Prove that these properties are equivalent, respectively, to all eigenvalues being nonnegative and all eigenvalues being strictly positive. Explain the role of a zero eigenvalue.

Solution · S02: prove necessity and sufficiency

If $q_i$ is a unit eigenvector, $q_i^TAq_i=\lambda_i$. Testing these vectors proves the required sign of every eigenvalue under either definiteness assumption. Conversely, in orthogonal coordinates,

$$x^TAx=\sum_i\lambda_i y_i^2.$$

If every $\lambda_i\ge0$, this is nonnegative for every $x$. If every $\lambda_i>0$ and $x\ne0$, at least one $y_i$ is nonzero, so the sum is strictly positive. If a zero eigenvalue occurs, its nonzero eigenvector has quadratic value zero, excluding positive definiteness. When all other eigenvalues are nonnegative, the matrix remains positive semidefinite; its zero-value vectors are exactly its kernel, since a sum of nonnegative terms vanishes precisely when every coordinate belonging to a positive eigenvalue vanishes. No corresponding sign criterion is asserted here for a nonsymmetric matrix using its eigenvalues alone.

Exercise S03: choose a basis inside a repeated eigenspace

For $B=\operatorname{diag}(2,2,-1)$, find an orthonormal eigenbasis containing $(1,1,0)^T/\sqrt2$. Give the associated factorization and all unit vectors attaining the maximum Rayleigh quotient. Are two arbitrary eigenvectors with eigenvalue two necessarily orthogonal?

Solution · S03: repeated eigenvalues leave a choice

The eigenspace for two is $\{(s,t,0)^T:s,t\in\mathbb R\}$; the eigenspace for minus one is the third coordinate axis. Choose

$$R=\begin{pmatrix}1/\sqrt2&1/\sqrt2&0\\1/\sqrt2&-1/\sqrt2&0\\0&0&1\end{pmatrix},\qquad D_B=\operatorname{diag}(2,2,-1).$$

Its columns have unit length and pairwise zero dot products, so $R^TR=I_3$. The first two columns are eigenvectors with eigenvalue two and the third has eigenvalue minus one. Therefore $BR=RD_B$ and $B=RD_BR^T$. On the unit sphere,

$$x^TBx=2(x_1^2+x_2^2)-x_3^2=2-3x_3^2.$$

The maximum is two, attained exactly when $x_3=0$ and $x_1^2+x_2^2=1$. Arbitrary vectors in the repeated eigenspace need not be orthogonal: $(1,0,0)^T$ and $(1,1,0)^T$ have dot product one. The spectral theorem guarantees a choice of orthonormal eigenbasis, not orthogonality of every possible list of eigenvectors.