4. Inner products and orthogonal projection

All spaces are finite-dimensional over $\mathbb R$ or $\mathbb C$.

Throughout the inner-product units $F=\mathbb R$ or $\mathbb C$, and $\langle u,v\rangle$ is linear in its first argument and conjugate-linear in its second. A different convention reverses some coefficients, so fix one before computing.

Definition

Inner product and orthogonality

An inner product satisfies conjugate symmetry $\langle u,v\rangle=\overline{\langle v,u\rangle}$, linearity in the first argument, and $\langle v,v\rangle>0$ for $v\ne0$. Put $\|v\|=\sqrt{\langle v,v\rangle}$. Vectors are orthogonal if their inner product is zero. For $U\subseteq V$, $U^\perp=\{v:\langle v,u\rangle=0\ \forall u\in U\}$.

For $w\ne0$, put $\alpha=\langle v,w\rangle/\|w\|^2$. Expanding $\|v-\alpha w\|^2\ge0$ yields the Cauchy–Schwarz inequality

$$|\langle v,w\rangle|\le\|v\|\|w\|,$$

with equality exactly when $v,w$ are dependent. The triangle inequality follows by expanding $\|v+w\|^2$ and applying this bound to the cross term. Positive definiteness, not mere symmetry, is essential here.

Gram–Schmidt converts an independent list $v_1,\ldots,v_k$ to an orthonormal list spanning the same successive subspaces:

$$w_j=v_j-\sum_{i<j}\langle v_j,q_i\rangle q_i,\qquad q_j=w_j/\|w_j\|.$$

Each $w_j$ is orthogonal to previous $q_i$ and is nonzero because otherwise $v_j$ would lie in the span of its predecessors. A zero or dependent input requires omitting a vector rather than dividing by zero.

In the following expanded calculation, $e_i$ denotes the orthonormal vector called $q_i$ in the shared argument.

Proof

The two arguments

If $v\ne0$, nonnegativity of $\|u-\langle u,v\rangle v/\|v\|^2\|^2$ gives Cauchy–Schwarz; equality occurs exactly when the residual is zero. If $v=0$, equality and dependence are immediate. In Gram–Schmidt, each residual is orthogonal to all earlier $e_i$. It cannot be zero, since $v_j$ is not in the span of the preceding vectors. Normalization yields an orthonormal vector without changing the successive span. Induction completes the proof.

In the case $v\ne0$, put $c=\langle u,v\rangle/\|v\|^2$. Conjugate symmetry and linearity in the first argument give the explicit expansion

$$\|u-cv\|^2=\|u\|^2-\overline c\langle u,v\rangle-c\overline{\langle u,v\rangle}+|c|^2\|v\|^2 =\|u\|^2-\frac{|\langle u,v\rangle|^2}{\|v\|^2}.$$

Nonnegativity proves the inequality, and equality is equivalent to $u=cv$. For Gram–Schmidt, each $\ell<j$ satisfies

$$\langle w_j,e_\ell\rangle=\langle v_j,e_\ell\rangle-\sum_{i<j}\langle v_j,e_i\rangle\langle e_i,e_\ell\rangle=0.$$

The independent-input hypothesis excludes a zero residual, so every displayed denominator is positive.

For a subspace $U$ with orthonormal basis $q_1,\ldots,q_k$, define

$$P_Uv=\sum_{j=1}^k\langle v,q_j\rangle q_j.$$

Then $P_Uv\in U$ and $v-P_Uv\in U^\perp$, so $V=U\oplus U^\perp$ and the projection is independent of the chosen orthonormal basis. For every $u\in U$,

$$\|v-u\|^2=\|v-P_Uv\|^2+\|P_Uv-u\|^2,$$

proving $P_Uv$ is the unique closest point. Orthogonality gives both existence and optimality.

Worked example

Project onto a plane

In $\mathbb R^3$ let $U=\operatorname{span}\{(1,1,0),(1,-1,0)\}$. These vectors are orthogonal and both have norm $\sqrt2$. Thus for $v=(2,3,4)$ the projection is $(2,3,0)$ and the residual is $(0,0,4)$, whose length $4$ is the distance to $U$.

Choosing a method and avoiding mistakes

Use Gram–Schmidt when the given independent basis is not orthogonal; use projection directly once an orthonormal basis is known. For merely orthogonal non-unit $u_i$, divide each coefficient by $\|u_i\|^2$. Do not omit conjugation in a complex inner product or apply the finite-dimensional complement theorem to an arbitrary infinite-dimensional subspace without additional hypotheses.

Exercises

1. Apply Gram–Schmidt to $(1,1)$, $(1,0)$ in $\mathbb R^2$ in that order.
2. Find the nearest point to $(1,2,3)$ in $U=\operatorname{span}\{(1,1,1)\}$ and its distance.
3. For $z,w\in\mathbb C^2$ use $\langle z,w\rangle=z_1\overline{w_1}+z_2\overline{w_2}$. Compute the projection of $(i,1)$ onto $\operatorname{span}\{(1,i)\}$.

Full solutions

1. $e_1=(1,1)/\sqrt2$. Subtracting the first component from $(1,0)$ gives $w_2=(1/2,-1/2)$, so $e_2=(1,-1)/\sqrt2$.
2. The coefficient on $(1,1,1)$ is $(1+2+3)/3=2$, so the nearest point is $(2,2,2)$. The residual $(-1,0,1)$ has length $\sqrt2$.
3. The spanning vector has squared norm $2$, and $\langle(i,1),(1,i)\rangle=i-i=0$. The projection is therefore zero.

Connection

Orthogonal projection is a self-adjoint operator. Adjoints generalize the transpose and express how a map interacts with the inner product.