Spaces identifiability and least squares

Exact accounting identities are useful, but recorded costs can contain measurement error or omitted influences. Instead of forcing incompatible observations into an exact model, we can ask which model prediction is closest to them. Two questions must remain separate: whether the coefficients are identifiable, and whether the fitted predictions are accurate.

1 Span basis and a linear model

A subspace of $\mathbb R^m$ contains zero and is closed under vector addition and real scalar multiplication. The span of vectors is the set of their linear combinations. Vectors are linearly independent if a zero linear combination forces every coefficient to be zero. A basis is an independent spanning list; its size is the dimension.

For $A\in\mathbb R^{m\times n}$, the map $T(x)=Ax$ is linear because $T(cx+dy)=cT(x)+dT(y)$. Its image is the column space $\operatorname{col}A$, the span of the columns. The model can predict exactly the vectors in this subspace. Its rank is $\dim\operatorname{col}A$.

Finite basis existence and extension, without a counting assumption

Let $V$ be a real vector space supplied with a finite spanning list $t_1,\ldots,t_N$. First prove that every independent list $u_1,\ldots,u_k$ has length at most $N$. After $j$ insertions, maintain a spanning list consisting of $u_1,\ldots,u_j$ and $N-j$ surviving old vectors. Expand $u_{j+1}$ in this list. Some coefficient on a surviving old vector is nonzero; otherwise $u_{j+1}$ would be in the span of the earlier $u_i$, contradicting independence. Solve for that old vector and replace it by $u_{j+1}$. The new list spans the same space and keeps every earlier insertion. If $k>N$, after $N$ steps the first $N$ independent vectors already span $V$, contradicting independence of $u_{N+1}$. This proves the bound, including $N=0$.

Every subspace $W$ has a finite basis. Start with the empty independent list. If its current span is not $W$, choose one vector of $W$ outside that span. A relation with nonzero coefficient on the new vector would express it in the old span, so the enlarged list remains independent. The proved bound permits at most $N$ additions; therefore the process stops with an independent spanning list for $W$. If $W=\{0\}$, it stops immediately with the empty basis. Starting instead with any given independent list in $W$ proves that list extends to a basis of $W$. Only a bounded finite selection is used. Finally, if two bases have sizes $p,q$, use each as the finite spanning list in the bound to obtain $p\le q$ and $q\le p$. Thus basis size, called dimension, is well-defined. This supplies the existence, extension and dimension facts used in rank–nullity.

Apply the following result with $V=\mathbb R^n$, $W=\mathbb R^m$ and $T(x)=Ax$.

A map $T:V\to W$ is linear if $T(\alpha v+\beta u)=\alpha Tv+\beta Tu$. Its kernel and image are subspaces. Choose a basis $k_1,\ldots,k_r$ of $\ker T$ and extend to a basis with $v_1,\ldots,v_s$. Then $Tv_1,\ldots,Tv_s$ form a basis of $\operatorname{im}T$: spanning follows by expanding any input; a dependence among these images places the corresponding input combination in the kernel, and independence of the extended basis makes all coefficients zero. Thus

$$\dim V=\dim\ker T+\dim\operatorname{im}T.$$

This is rank–nullity. In equal finite dimensions, injectivity and surjectivity are equivalent, but the equivalence can fail in infinite-dimensional spaces.

A coefficient vector is identifiable from its exact prediction precisely when $\ker A=\{0\}$. For example, if two explanatory columns are identical, their coefficients can trade off invisibly. More observations do not fix this when they repeat the same dependence.

Real inner products, their norms and the needed inequalities

An inner product on a real vector space is a map $\langle u,v\rangle\in\mathbb R$ that is symmetric, linear in each argument, and positive definite: $\langle v,v\rangle>0$ for $v\ne0$. Linearity gives $\langle0,v\rangle=0$. Define $\|v\|=\sqrt{\langle v,v\rangle}$. It is zero exactly at zero and satisfies $\|cv\|=|c|\|v\|$. The standard example on $\mathbb R^m$ is $\langle u,v\rangle=\sum_{i=1}^mu_iv_i$. Vectors are orthogonal when their inner product is zero, and a list is orthonormal when distinct members are orthogonal and each member has norm one.

If $v\ne0$, insert $t=\langle u,v\rangle/\|v\|^2$ into the nonnegative square $\|u-tv\|^2$. Expanding by bilinearity gives $\|u\|^2-\langle u,v\rangle^2/\|v\|^2\ge0$, hence $|\langle u,v\rangle|\le\|u\|\|v\|$. If $v=0$ the same inequality holds directly. Equality for $v\ne0$ means $\|u-tv\|=0$, equivalently $u=tv$; the zero-vector cases also give equality. Finally expand $\|u+v\|^2$ and apply the bound to its cross term to obtain $\|u+v\|^2\le(\|u\|+\|v\|)^2$. Both sides to be square-rooted are nonnegative, so the triangle inequality follows. Thus this is a norm, and normalization of any nonzero vector divides by a positive real number. No complex bilinear convention is being used.

Orthonormal bases and extension while retaining prescribed vectors

Suppose in addition that $V$ has a real positive-definite inner product. On an independent list $v_1,\ldots,v_s$, define successively

$$w_j=v_j-\sum_{i<j}\langle v_j,q_i\rangle q_i,\qquad q_j=w_j/\|w_j\|.$$

Inductively the previous $q_i$ are orthonormal. For $l<j$, taking the inner product with $q_l$ gives $\langle w_j,q_l\rangle=\langle v_j,q_l\rangle-\langle v_j,q_l\rangle=0$. If $w_j=0$, then $v_j$ lies in the previous span, a contradiction. Division by its positive norm is therefore legal. The displayed formula and its rearrangement show that the first $j$ old and new vectors span the same space, completing the induction. Apply this to a basis to obtain an orthonormal basis, also for a subspace. To extend a prescribed orthonormal list, first extend it to an ordinary basis by the proved theorem and then apply this construction in that order. On the prescribed first vectors no component is subtracted and the norm is one, so every prescribed vector is retained exactly. Empty lists and the zero subspace require no division. This proves the orthonormal extension used in SVD and projection existence.

2 Orthogonal residuals and normal equations

The Euclidean inner product is $u^Tv=\sum u_iv_i$, and $\|u\|^2=u^Tu$. A least-squares solution minimizes $\|Ax-b\|^2$ over all real $x$. Its residual is $r=b-Ax$.

Theorem. A vector $\hat x$ is a least-squares solution if and only if

$$A^T(A\hat x-b)=0.$$

These are the normal equations. If the columns of $A$ are independent, the solution is unique.

Proof. For any perturbation $h$,

$$\|A(\hat x+h)-b\|^2=\|A\hat x-b\|^2+2h^TA^T(A\hat x-b)+\|Ah\|^2.$$

If the normal equations hold, the last term is nonnegative, proving minimality. Conversely, take $h=te_j$. A nonzero linear coefficient in this quadratic would allow a sufficiently small choice of $t$ of the opposite sign to decrease the objective. Thus all coordinates of $A^T(A\hat x-b)$ vanish. If columns are independent, $h^TA^TAh=\|Ah\|^2>0$ for $h\ne0$, so $A^TA$ is invertible and the normal equations have one solution. $\square$

Existence does not require independent columns: choose an orthonormal basis of the finite-dimensional column space, project $b$ onto that basis, and choose any coefficient vector representing that projection. Predictions are unique even when coefficients are not.

3 Worked cost estimation

Fit the simplified model $c=\alpha+\beta q$ to three observations $(q,c)=(0,2),(1,3),(2,5)$. The intercept models a fixed component and the slope a variable component, but those interpretations require the model to be appropriate.

$$A=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix},\quad b=\begin{pmatrix}2\\3\\5\end{pmatrix},\quad A^TA=\begin{pmatrix}3&3\\3&5\end{pmatrix},\quad A^Tb=\binom{10}{13}.$$

Subtracting the first normal equation from the second gives $2\beta=3$. Hence $\beta=3/2$ and $\alpha=11/6$. Predictions are $(11/6,10/3,29/6)$ and the residual is $(1/6,-1/3,1/6)$. Its sum and its dot product with $(0,1,2)$ are both zero, as the theorem predicts. Squared residual norm is $1/6$.

A small residual does not prove causation, validate future extrapolation, or detect all omitted variables. A negative estimated fixed cost may indicate an unsuitable model or unsuitable data; unconstrained least squares does not enforce accounting constraints automatically. For numerical computation, QR or SVD avoids the conditioning penalty of explicitly forming $A^TA$.

4 Graded practice

E1 Foundation. Show $W=\{(x,y,z):x+y-z=0\}$ is a subspace and find a basis.

E2 Computation. Fit $c=\alpha+\beta q$ to $(0,1),(1,2),(2,2)$. Find the residual and its squared norm.

E3 Structure. If $A$ has columns $u,u$ with $u\ne0$, describe all coefficients minimizing $\|Ax-b\|^2$.

E4 Extension. Let $D=\operatorname{diag}(d_i)$ have nonzero real diagonal entries and write the statistical weights as $w_i=d_i^2>0$. Derive the equations for minimizing $\|D(Ax-b)\|^2$. Explain why the $w_i$ must be positive for a genuine weighted norm, while choosing $d_i>0$ is a square-root convention and nonzero $d_i$ of either sign give the same weights.

5 Full solutions

S1. The defining equation is homogeneous linear, so zero lies in $W$ and linear combinations preserve it. Every vector is $(x,y,x+y)=x(1,0,1)+y(0,1,1)$. These two vectors are independent by their first two coordinates, hence form a basis.

S2. Here $A^Tb=(5,6)^T$ with the same $A^TA$ as above. Subtraction gives $2\beta=1$, so $\beta=1/2$ and $\alpha=7/6$. The residual is $(-1/6,1/3,-1/6)$, squared norm $1/6$. It is perpendicular to both model columns.

S3. The prediction is $(x_1+x_2)u$. Projection onto the line spanned by $u$ gives the unique sum $s=u^Tb/(u^Tu)$. Every $(t,s-t)$ minimizes the objective. The null direction $(1,-1)$ explains coefficient nonuniqueness.

S4. Apply the normal-equation theorem to $DA$ and $Db$: $A^TD^2(Ax-b)=0$. Positive diagonal entries make $D$ invertible, so $\|Dv\|=0$ only when $v=0$. A zero weight can erase a nonzero observation difference; a negative diagonal entry still gives the same squared positive weight after squaring, but is not the conventional positive square-root weight. The statistical meaning of a chosen weight requires separate assumptions.

Summary

Column spaces describe possible predictions; null spaces describe nonidentifiability. Least squares projects an observation vector onto the prediction space. Orthogonality produces the normal equations, but a good fit is a mathematical property of a chosen model, not a guarantee of business validity.