1.1 Vectors, inner products and distance

From positions to displacements

A temperature field takes a position as input. A particle trajectory returns a position as output. Before differentiating either object, we need a language for comparing positions and describing directions. Coordinates supply the algebra; the inner product supplies length and angle.

Definition

Euclidean space and vectors

The space $\mathbb R^n$ consists of ordered tuples $x=(x_1,\ldots,x_n)$ of real numbers. Addition and scalar multiplication act component by component:

$$x+y=(x_1+y_1,\ldots,x_n+y_n),\qquad cx=(cx_1,\ldots,cx_n).$$

The standard basis vector $e_i$ has entry $1$ in position $i$ and $0$ elsewhere, so $x=\sum_{i=1}^n x_i e_i$. A point specifies a position, whereas a vector specifies a displacement. For points $P,Q$, the vector from $P$ to $Q$ is $Q-P$.

A linear combination $a_1v_1+\cdots+a_kv_k$ combines displacement vectors. An affine combination of points has coefficients summing to $1$. This condition makes the resulting point independent of the chosen origin. In particular, $(1-t)P+tQ$ describes the segment from $P$ to $Q$ when $0\le t\le1$; allowing all real $t$ describes the full line.

Worked example

A displacement and its direction

Take $P=(-2,1,0)$ and $Q=(1,5,0)$. Then $Q-P=(3,4,0)$. Its length is $5$, so the unit vector pointing from $P$ to $Q$ is $(3/5,4/5,0)$. The point one quarter of the way along the segment is

$$P+\tfrac14(Q-P)=(-5/4,2,0).$$

Changing the origin changes the coordinates of both points, but their difference and the segment parameter remain the same.

Length and angle from the inner product

Definition

Inner product, norm and distance

For $x,y\in\mathbb R^n$, define

$$\langle x,y\rangle=\sum_{i=1}^n x_i y_i, \qquad \|x\|=\sqrt{\langle x,x\rangle}, \qquad d(P,Q)=\|Q-P\|.$$

The inner product is a scalar. It is symmetric and linear in each variable. Also $\langle x,x\rangle\ge0$, with equality exactly when $x=0$. Consequently $\|cx\|=|c|\|x\|$.

Two vectors are orthogonal when their inner product is zero. For nonzero $x,y$, their angle $\theta\in[0,\pi]$ satisfies

$$\cos\theta=\frac{\langle x,y\rangle}{\|x\|\|y\|}.$$

The inequality below ensures that this quotient lies in $[-1,1]$. The zero vector is orthogonal to every vector, but has no angle or unit direction.

Theorem

Cauchy–Schwarz inequality

For $x,y\in\mathbb R^n$,

$$|\langle x,y\rangle|\le\|x\|\|y\|.$$

Equality holds exactly when the pair is linearly dependent, including pairs containing a zero vector.

Proof

Proof by removing a parallel component

If $y=0$, both sides vanish. If $y\ne0$, put $c=\langle x,y\rangle/\|y\|^2$. Nonnegativity gives

$$0\le\|x-cy\|^2 =\|x\|^2-\frac{\langle x,y\rangle^2}{\|y\|^2}.$$

Multiply by $\|y\|^2$ and take square roots. Equality means $x-cy=0$, precisely the required dependence condition when $y\ne0$.

Worked example

A linear expression on a sphere

Suppose $x^2+y^2+z^2=4$. The expression $3x-4y+12z$ is the inner product of $(3,-4,12)$ with $(x,y,z)$. Their lengths are $13$ and $2$, so

$$|3x-4y+12z|\le26.$$

Equality requires parallel vectors. The maximum $26$ occurs at $(6,-8,24)/13$, and the minimum $-26$ occurs at its negative. A bound becomes a complete extremum calculation only after these equality points are identified.

Estimates that control distance

Theorem

Triangle and reverse triangle inequalities

For all $x,y\in\mathbb R^n$,

$$\|x+y\|\le\|x\|+\|y\|, \qquad \big|\|x\|-\|y\|\big|\le\|x-y\|.$$

Equality in the first inequality holds if one vector is zero or the two nonzero vectors point in the same direction.

Proof

Proof and equality conditions

Expand the square and apply Cauchy–Schwarz:

$$\|x+y\|^2 =\|x\|^2+2\langle x,y\rangle+\|y\|^2 \le(\|x\|+\|y\|)^2.$$

Taking square roots is valid because both sides are nonnegative. For nonzero vectors, equality requires $\langle x,y\rangle=\|x\|\|y\|$, which is positive dependence, rather than arbitrary dependence. Finally, apply the triangle inequality to $x=(x-y)+y$, then exchange $x$ and $y$. The two resulting bounds give the reverse triangle inequality.

The reverse inequality says that moving a point by at most $\varepsilon$ changes its distance from the origin by at most $\varepsilon$. This is a first example of an estimate that controls every possible direction of approach, a central requirement in multivariable limits.

Orthogonal projection

We often need to split a displacement into a part along a chosen direction and a part perpendicular to it. For a nonzero vector $v$, requiring $x-cv$ to be orthogonal to $v$ gives $\langle x,v\rangle-c\|v\|^2=0$. Hence

$$p=\operatorname{proj}_v x=\frac{\langle x,v\rangle}{\|v\|^2}v, \qquad x=p+(x-p),\quad \langle x-p,v\rangle=0.$$

This is also a nearest-point construction. Write $p=cv$. For every real $t$, orthogonality gives

$$\|x-tv\|^2=\|x-p\|^2+(c-t)^2\|v\|^2.$$

The unique minimum occurs at $t=c$. Thus projection solves a minimization problem without differentiation.

Worked example

Check a projection by its residual

For $x=(4,1,2)$ and $v=(1,1,0)$, the coefficient is $5/2$. Therefore

$$p=(5/2,5/2,0),\qquad x-p=(3/2,-3/2,2).$$

The residual has dot product $0$ with $v$. Also $\|p\|^2=25/2$ and $\|x-p\|^2=17/2$, whose sum is $21=\|x\|^2$. Both checks confirm that the decomposition is orthogonal.

Common mistake

A projection is a vector

The coefficient $\langle x,v\rangle/\|v\|^2$ is a scalar; multiply it by $v$ to obtain the projection. A negative coefficient means the projection points opposite to $v$. A projection's norm cannot exceed $\|x\|$, although an individual coordinate can increase.

How an algebraic formula becomes geometry

The angle formula agrees with the ordinary angle between two nonzero arrows in the plane they span. To see this, expand the squared length of their difference:

$$\|x-y\|^2=\|x\|^2+\|y\|^2-2\langle x,y\rangle.$$

The cosine law for the triangle with vertices $0,x,y$ gives the same expression with the last term replaced by $-2\|x\|\|y\|\cos\theta$. Comparing the two identifies the quotient defining the angle. Cauchy–Schwarz establishes that the algebraic definition works in every dimension, even when the entire ambient space cannot be drawn.

When $\langle x,y\rangle=0$, the cross term vanishes and we recover Pythagoras. Conversely, if $\|x+y\|^2=\|x\|^2+\|y\|^2$, then expansion forces $\langle x,y\rangle=0$. Thus a length identity can detect orthogonality without computing an angle. More generally, the sign of the inner product distinguishes an angle below, equal to, or above $\pi/2$, provided both vectors are nonzero. Parallel vectors can have angle $0$ or $\pi$, depending on whether they point in the same or opposite directions.

The squared norm often makes calculations shorter because it is a polynomial expression. For instance, checking $\|u+v\|=\|u-v\|$ is equivalent to checking equality of their squares, since norms are nonnegative. Subtracting the expansions gives $4\langle u,v\rangle=0$. This proves that the diagonals corresponding to $u+v$ and $u-v$ have equal lengths exactly when $u$ and $v$ are orthogonal.

Use estimates rather than checking directions

The triangle inequality is more than a statement about triangles. For points $P,Q,R$, write $R-P=(Q-P)+(R-Q)$ to obtain

$$d(P,R)\le d(P,Q)+d(Q,R).$$

It bounds the total effect of two consecutive errors. If a measured position differs from an intermediate estimate by at most $a$, and that estimate differs from the true position by at most $b$, then the original measurement differs from the truth by at most $a+b$. No assumption about the directions of the two errors is needed.

Cauchy–Schwarz provides a similarly uniform estimate for a fixed linear measurement. With $a\in\mathbb R^n$ fixed, put $\ell(x)=\langle a,x\rangle$. Then

$$|\ell(x)-\ell(y)|=|\langle a,x-y\rangle| \le\|a\|\|x-y\|.$$

For example, if $a=(2,-1,2)$ and a position changes by at most $0.01$ in Euclidean distance, the measurement $2x_1-x_2+2x_3$ changes by at most $0.03$. The estimate covers all changes in position simultaneously. Equality is possible when the displacement is parallel to $a$, so the constant $3$ cannot be made smaller for all possible displacements. This gives both an error bound and the direction in which the measurement is most sensitive.

Projection also gives a uniform comparison. If $p=\operatorname{proj}_v x$, then Pythagoras yields $\|x\|^2=\|p\|^2+\|x-p\|^2$, hence $\|p\|\le\|x\|$. Equality holds precisely when the residual vanishes, meaning $x$ already lies on the line spanned by $v$. Applying the projection twice changes nothing: writing $p=cv$ gives $\operatorname{proj}_v p=cv=p$. These two properties explain why projection behaves like retaining one directional component and discarding the perpendicular component.

A problem-solving routine

First identify the type of each object. The difference of two positions is a vector; its norm is a length; an inner product produces a scalar; a projection produces a vector. Keeping these types visible prevents a scalar projection coefficient from being mistaken for the projected displacement.

Next select the quantity that matches the question. For a distance, form a difference and take its norm. For an angle, compute an inner product and both lengths. For an upper bound on a linear expression, recognize an inner product and apply Cauchy–Schwarz before doing more coordinate algebra. For the closest point along one direction, remove the perpendicular residual using projection.

Finally verify the result geometrically. A unit vector must have norm $1$ and the intended direction. A projection residual must be orthogonal to the target direction. An extremum found from an inequality must satisfy the original constraint and the inequality's equality condition. These checks have different purposes: one confirms magnitude, another confirms direction, and another confirms attainability. Together they convert a plausible calculation into a complete solution.

Coordinates, scaling and geometric meaning

The standard basis separates a vector into independent coordinate contributions. The coefficient of a basis vector tells us how much displacement occurs in that coordinate direction. These coefficients can be negative, positive or zero. A negative coefficient describes movement opposite to a basis direction; it does not describe a negative length. This distinction explains why coordinate calculations use signed numbers while a norm uses a square root of a sum of squares. The sign information remains in the vector even though it disappears from its length.

To compare vectors, first check that they belong to the same Euclidean space. A point in the plane and a point in three-dimensional space do not have a difference until an embedding has been specified. Once the ambient space is fixed, the formulas do not depend on whether we can draw that space. In four dimensions, for example, two positions differing only in one coordinate are separated by the absolute value of that coordinate change. When several coordinates change, the squared changes add, just as they do in a right triangle in the plane.

The dot product records how two displacements cooperate. Positive contributions from corresponding coordinates can be cancelled by negative contributions in other coordinates. Orthogonality therefore does not require the nonzero coordinates to occur in different positions. For instance, the planar vectors with coordinates one and two, and coordinates two and negative one, have two nonzero coordinates each, but their inner product is zero. The cancellation expresses perpendicular directions rather than a lack of shared coordinates. This is why inspecting coordinates one by one is usually less useful than computing the full inner product.

Scaling a direction changes a parameter but does not change its line. If a nonzero real number multiplies the target vector in a projection formula, it appears once in the inner product, twice in the squared denominator and once in the final vector. These factors cancel. Thus projecting onto a direction and projecting onto any nonzero multiple of that direction give the same point. In particular, reversing the arrow chosen to describe the line leaves the nearest point unchanged. This is a valuable consistency check when two solutions use different direction vectors.

The signed scalar component along a unit direction deserves separate attention. It is an inner product and may be positive, zero or negative. The corresponding projected vector is that scalar times the unit direction. Its length is the absolute value of the scalar. Confusing these three objects can produce an answer of the wrong type even when all arithmetic is correct. In a displacement problem, retain the sign when describing which way an object moves; take the absolute value only when the question asks for a distance or length.

Finally, inequalities and equations play different roles. An inequality such as Cauchy–Schwarz provides a bound valid for an entire set of vectors. An equation describing the equality case identifies the vectors for which the bound is sharp. In a constrained extremum problem, both pieces are needed. If the equality direction is incompatible with an extra constraint, the original bound remains true but may no longer be attainable. The correct next step is then to examine the additional constraint, rather than to label the unattained bound as a maximum. Keeping the bound and its attainment separate is a habit that will remain useful in later optimization problems.

Exercises and solutions

Checkpoint

1. Expand both sides to prove the parallelogram identity.

Prove $\|x+y\|^2+\|x-y\|^2=2\|x\|^2+2\|y\|^2$.

Solution · 1. The cross terms cancel

The two expansions contain $+2\langle x,y\rangle$ and $-2\langle x,y\rangle$. Their sum is $2\langle x,x\rangle+2\langle y,y\rangle$, giving the identity.

Checkpoint

2. Compare the two equality conditions.

Let $u=(2,-1,2)$ and $v=(-4,2,-4)$. Does equality hold in Cauchy–Schwarz? Does it hold in $\|u+v\|\le\|u\|+\|v\|$?

Solution · 2. Opposite directions give different answers

Since $v=-2u$, Cauchy–Schwarz is an equality: $|\langle u,v\rangle|=18=3\cdot6$. But $u+v=-u$, so the triangle inequality reads $3\lt9$ and is strict. Opposite directions satisfy dependence, but not positive dependence.

Checkpoint

3. Find a projection and its error.

Project $x=(2,5)$ onto $v=(1,2)$, and find the shortest distance from $x$ to the line through $0$ in direction $v$.

Solution · 3. Use the perpendicular residual

Here $\langle x,v\rangle=12$ and $\|v\|^2=5$, giving $p=(12/5,24/5)$. The residual $(-2/5,1/5)$ is perpendicular to $v$, and its length is $\sqrt5/5$. The nearest-point identity proves that this is the shortest distance.