Partial derivatives and local models

With the matrix notation in place, we distinguish coordinate partial derivatives from total differentiability, then use the chain rule before turning to applications.

Here $e_i$ is the standard basis vector with $1$ in position $i$ and $0$ elsewhere; Euclidean norms and linear maps preserving addition and scalar multiplication are assumed preparation, with refreshers in vectors and norms and linear maps and linearization.

Matrix notation and products

Definition

When a matrix product is defined

If $A$ is an $m \times n$ matrix and $B$ is an $n \times p$ matrix, then the product $AB$ is defined and is an $m \times p$ matrix.

If the number of columns of $A$ does not equal the number of rows of $B$, then the product $AB$ is undefined.

Definition

Matrix multiplication

Suppose $A = [a_{ij}]$ is an $m \times n$ matrix and $B = [b_{jk}]$ is an $n \times p$ matrix.

Then the $(i,k)$ entry of $AB$ is

$$(AB)_{ik} = a_{i1}b_{1k} + a_{i2}b_{2k} + \cdots + a_{in}b_{nk}.$$

So each output entry is the dot-product-style combination of row $i$ of $A$ with column $k$ of $B$.

Identity matrices do nothing, on purpose

Definition

Identity matrix

For each positive integer $n$, the identity matrix $I_n$ is the $n \times n$ square matrix with $1$ on the main diagonal and $0$ everywhere else.

For example,

$$I_2 = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}, \qquad I_3 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}.$$

The identity matrix matters because it preserves any compatible matrix:

$$AI_n = A, \qquad I_m A = A$$

whenever the sizes match.

Worked example

Why multiplying by the identity changes nothing

Let

$$A = \begin{bmatrix} 2 & -1 \\ 4 & 3 \end{bmatrix}.$$

Then

$$AI_2 = \begin{bmatrix} 2 & -1 \\ 4 & 3 \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 2 & -1 \\ 4 & 3 \end{bmatrix}.$$

The first column of $AI_2$ reproduces the first column of $A$, and the second column reproduces the second column of $A$.

Counterexample mode

Matching dimensions do not guarantee commutativity

The claim “two square matrices of the same size always satisfy $AB=BA$” is false even though both products exist and have the same size. Take

$$A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\qquad B=\begin{bmatrix}0&0\\0&1\end{bmatrix}.$$

Both are real $2\times2$ matrices, so the hypothesis is satisfied. Direct row-by-column multiplication gives

$$AB=\begin{bmatrix}0&2\\0&4\end{bmatrix},\qquad BA=\begin{bmatrix}0&0\\3&4\end{bmatrix}.$$

For example, $(AB)_{12}=1\cdot0+2\cdot1=2$, whereas $(BA)_{12}=0\cdot2+0\cdot4=0$. One unequal corresponding entry is enough to prove $AB\ne BA$. The failure has a structural explanation: multiplying this $B$ on the right keeps the second column and zeros the first, while multiplying it on the left keeps the second row and zeros the first.

A valid replacement is: a square matrix commutes with every scalar multiple of the identity of the same size. Indeed, if $B=cI$, then $AB=A(cI)=cA=(cI)A=BA$. For arbitrary square matrices, commutativity needs its own justification; matching dimensions guarantees that multiplication is defined, not that factors can be interchanged.

Matrix multiplication represents composition

The multiplication rule is not arbitrary. It is the rule that makes matrices encode linear transformations in sequence.

If a vector $x$ is first sent to $Bx$, and then that result is sent to $A(Bx)$, the combined effect is

$$(AB)x.$$

That is why the inner dimensions must match. The output of the first map must be a valid input for the second one.

Theorem

Associativity matches repeated composition

Whenever the products are defined,

$$A(BC) = (AB)C.$$

So we may regroup a chain of matrix products without changing the final linear transformation.

This does not mean that order may be changed. Associativity lets us change parentheses, not the order of the factors themselves.

Definition

Transpose

If $A = [a_{ij}]$ is an $m \times n$ matrix, then its transpose $A^T$ is the $n \times m$ matrix whose $(j,i)$ entry is $a_{ij}$.

Equivalently,

$$(A^T)_{ji} = a_{ij}.$$

So every row of $A$ becomes a column of $A^T$, and every column of $A$ becomes a row.

Theorem

Basic properties of transpose

Let $A$ and $B$ have the same size when they are added, let $c$ be a scalar, and suppose $A$ is $m \times n$ and $B$ is $n \times p$ when they are multiplied. Then:

1. $(A^T)^T = A$
2. $(A + B)^T = A^T + B^T$
3. $(cA)^T = cA^T$
4. $(AB)^T = B^T A^T$

Proof

Entrywise proof of the product rule

Let $A=[a_{ij}]$ be $m \times n$ and $B=[b_{ij}]$ be $n \times p$. Then $AB$ is $m \times p$, so both $(AB)^T$ and $B^T A^T$ are $p \times m$. For $1 \le i \le p$ and $1 \le j \le m$, the $(i,j)$ entry of the first matrix is

$$\bigl((AB)^T\bigr)_{ij} =(AB)_{ji} =\sum_{k=1}^{n} a_{jk}b_{ki}.$$

On the other hand, the $(i,j)$ entry of $B^T A^T$ is

$$\begin{aligned} (B^T A^T)_{ij} &=\sum_{k=1}^{n}(B^T)_{ik}(A^T)_{kj} \\ &=\sum_{k=1}^{n}b_{ki}a_{jk} \\ &=\sum_{k=1}^{n}a_{jk}b_{ki}. \end{aligned}$$

The last equality uses commutativity of scalar multiplication, not commutativity of matrix multiplication. Corresponding entries therefore agree, so $(AB)^T=B^T A^T$. The dimensions also explain the reversed order: $B^T$ is $p \times n$ and $A^T$ is $n \times m$, whereas $A^T B^T$ would generally not even be defined.

Coordinate partial derivatives

Definition

Partial derivative at a point

Let $U\subseteq\mathbb R^n$ be open, let $f:U\to\mathbb R$, and let $a\in U$. The $i$th partial derivative is the limit

$$\partial_i f(a)= \lim_{t\to0}\frac{f(a+t e_i)-f(a)}{t},$$

when that finite limit exists. We also write $\frac{\partial f}{\partial x_i}(a)$. For two variables, $\partial_1f=f_x$ and $\partial_2f=f_y$.

Definition

Gradient and repeated partial derivatives

When all first partial derivatives exist at $a$, the gradient is

$$\nabla f(a)=\bigl(\partial_1f(a),\ldots,\partial_nf(a)\bigr).$$

We use the convention $f_{xy}=\partial_y(f_x)$: first differentiate with respect to $x$, then with respect to $y$. A function is of class $C^r$ on an open set if all partial derivatives of orders at most $r$ exist and are continuous there.

Theorem

Algebra of partial derivatives

If $f$ and $g$ have an $i$th partial derivative at $a$, then so do $f+g$ and $fg$, with

$$\partial_i(f+g)(a)=\partial_i f(a)+\partial_i g(a), \qquad \partial_i(fg)(a)=g(a)\partial_i f(a)+f(a)\partial_i g(a).$$

If also $g(a)\ne0$, the quotient is defined along the coordinate line sufficiently near $a$ and

$$\partial_i(f/g)(a) =\frac{g(a)\partial_i f(a)-f(a)\partial_i g(a)}{g(a)^2}.$$

Total differentiability and local linear models

Definition

Differentiability with a linear remainder

Let $U\subseteq\mathbb R^n$ be open and $F:U\to\mathbb R^m$. We say $F$ is differentiable at $a\in U$ if there is a linear map $A:\mathbb R^n\to\mathbb R^m$ such that

$$F(a+h)=F(a)+Ah+R(h), \qquad \lim_{h\to0,\ h\ne0}\frac{\|R(h)\|}{\|h\|}=0.$$

The map $A$ is the derivative, denoted $DF(a)$. The notation $R(h)=o(\|h\|)$ abbreviates the displayed limit.

Definition

Linearization and total differential

The linearization of $F$ at $a$ is the affine function

$$L_a(x)=F(a)+DF(a)(x-a).$$

For a scalar-valued differentiable function,

$$DF(a)h=\sum_{i=1}^n\partial_i f(a)h_i=\nabla f(a)\cdot h.$$

The first-order predicted change is called the total differential:

$$df_a=\sum_{i=1}^n\partial_i f(a)\,dx_i.$$

Theorem

Consequences and uniqueness of a derivative

If $F$ is differentiable at $a$, it is continuous at $a$. Its coordinate partial derivatives exist there and are the columns of $DF(a)$. The linear map satisfying the differentiability definition is unique.

Proof

Continuity, coordinate derivatives and uniqueness

A linear map in finite-dimensional Euclidean spaces has a bound $\|Ah\|\le C\|h\|$ for some constant $C$. The defining expansion therefore gives

$$\|F(a+h)-F(a)\|\le C\|h\|+\|R(h)\|\longrightarrow0.$$

Thus $F$ is continuous at $a$. Set $h=t e_i$ in the expansion and divide by $t$. The remainder tends to zero after division, so the $i$th coordinate derivative is $Ae_i$.

If both $A$ and $B$ satisfy the definition, subtraction gives $(A-B)h=o(\|h\|)$. Taking $h=t e_i$ shows $(A-B)e_i=0$ for every basis vector. A linear map is determined by its values on a basis, so $A=B$.

Theorem

Continuous first partial derivatives are sufficient

Let $f:U\to\mathbb R$ be defined on an open set. If all first partial derivatives exist on a neighbourhood of $a$ and are continuous at $a$, then $f$ is differentiable at $a$. In particular, every $C^1$ function on an open set is differentiable there. The same statement applies component by component to a vector-valued function.

Proof

Why continuous partial derivatives control every direction

For two variables, write the change from $(a,b)$ to $(a+h,b+k)$ as two coordinate changes:

$$f(a+h,b+k)-f(a,b) =[f(a+h,b+k)-f(a,b+k)] +[f(a,b+k)-f(a,b)].$$

The one-variable mean value theorem gives intermediate coordinates $\xi$ between $a$ and $a+h$, and $\eta$ between $b$ and $b+k$, so the change equals

$$f_x(\xi,b+k)h+f_y(a,\eta)k.$$

After subtracting $f_x(a,b)h+f_y(a,b)k$, divide the absolute remainder by $\sqrt{h^2+k^2}$. Since each coordinate increment has absolute value at most this norm, the ratio is at most

$$|f_x(\xi,b+k)-f_x(a,b)|+ |f_y(a,\eta)-f_y(a,b)|.$$

Both terms tend to zero by continuity. Zero coordinate increments simply omit the corresponding difference. In more variables, telescope along the coordinate directions in the same way.

Worked example

A continuous function with partial derivatives but no derivative

Define

$$g(x,y)= \begin{cases} \dfrac{xy}{\sqrt{x^2+y^2}},&(x,y)\ne(0,0),\\ 0,&(x,y)=(0,0). \end{cases}$$

The inequality $|xy|\le(x^2+y^2)/2$ gives $|g(x,y)|\le\sqrt{x^2+y^2}/2$, so $g$ is continuous at zero. Both coordinate restrictions vanish, hence both partial derivatives at zero are zero.

If a derivative existed, its coordinate columns would force it to be the zero map. Its remainder would then be $g$ itself. Along $(x,y)=(t,t)$,

$$\frac{|g(t,t)|}{\sqrt{t^2+t^2}}=\frac12$$

for every nonzero $t$. The required ratio does not tend to zero. Thus the function is continuous and has both coordinate derivatives, but is not differentiable at the origin.

The chain rule and rates of change

Definition

A derivative in a specified direction

Let $f$ be defined on an open neighbourhood of $a\in\mathbb R^n$, and let $u$ be a unit vector. The directional derivative is

$$D_uf(a)=\lim_{t\to0}\frac{f(a+tu)-f(a)}{t},$$

when the finite two-sided limit exists. Because $\|u\|=1$, the parameter $t$ measures signed distance along the line.

Theorem

Matrix chain rule

Let $F:U\subseteq\mathbb R^n\to\mathbb R^m$ be differentiable at $a$, and let $G:V\subseteq\mathbb R^m\to\mathbb R^p$ be differentiable at $F(a)$. Assume $U,V$ are open and the composite is defined near $a$. Then

$$D(G\circ F)(a)=DG(F(a))\,DF(a).$$

The matrix sizes are $(p\times m)(m\times n)=p\times n$.

Proof

Composing two linear approximations

Write $A=DF(a)$, $b=F(a)$ and $B=DG(b)$. For a small input displacement,

$$F(a+h)=b+Ah+r(h),\qquad r(h)=o(\|h\|).$$

Put $\Delta=Ah+r(h)$. The outer expansion is

$$G(b+\Delta)=G(b)+B\Delta+s(\Delta), \qquad s(\Delta)=o(\|\Delta\|).$$

Substitution gives the linear term $BAh$ and remainder $Br(h)+s(\Delta)$. A linear-map bound shows that $\|\Delta\|\le C\|h\|$ for all sufficiently small $h$. Consequently $Br(h)=o(\|h\|)$, and the outer remainder estimate gives $s(\Delta)=o(\|h\|)$ as well. At $\Delta=0$ this remainder is zero. Thus the composite has the required linear approximation with derivative $BA$.

Theorem

Rates along curves and the gradient

If $f$ is differentiable at $a=\gamma(t_0)$ and $\gamma$ is differentiable at $t_0$, then

$$(f\circ\gamma)'(t_0)=\nabla f(a)\cdot\gamma'(t_0).$$

In particular, for a unit vector $u$,

$$D_uf(a)=\nabla f(a)\cdot u.$$

If $\nabla f(a)\ne0$, the maximum unit directional rate is $\|\nabla f(a)\|$, attained in the direction $\nabla f(a)/\|\nabla f(a)\|$; the minimum is its negative, attained in the opposite direction.

For matrix calculations we write the gradient as a column. A scalar output has a one-row Jacobian:

$$Df(a)=\nabla f(a)^T,\qquad Df(a)h=\nabla f(a)^Th.$$

For $g:\mathbb R^n\to\mathbb R^m$, the Jacobian has $m$ output rows and $n$ input columns. This fixes the multiplication order in the chain rule; it does not change the partial derivatives.

Multivariable Sensitivity in Hypothetical Models

A scenario table is a linear map

For holdings $x_1,x_2$, suppose two specified scenarios give changes

$$\Delta v=\begin{pmatrix}2&-1\\-3&4\end{pmatrix}\begin{pmatrix}x_1\\x_2\end{pmatrix}.$$

At $(5,2)^T$ the changes are $(8,-7)^T$. Each column gives the two scenario responses to one unit of the corresponding holding. These are deterministic scenario outputs, not probabilities.

Discounted payment: domains, partials, and rate units

Let

$$P(F,r)=\frac{F}{1+r}.$$

The formula is mathematically defined for $r\ne-1$. Here the model takes $F\ge0$ and $r>-1$. Derivatives at an interior point use nearby inputs in every direction; at a boundary one must specify an extension or the relevant one-sided convention.

Holding the other input fixed gives

$$P_F=\frac1{1+r},\qquad P_r=-\frac{F}{(1+r)^2}.$$

At $F=110$, $r=0.1$, the value is $100$, with $P_F=10/11$ and $P_r=-1000/11$. The first partial is per payment unit; the second is per unit rate proportion. An increase of $0.01$ means one percentage point, not one percent of the existing rate.

For simultaneous changes $\Delta F=1$ and $\Delta r=0.001$, the first-order estimate is

$$\Delta P\approx\frac{10}{11}(1)-\frac{1000}{11}(0.001)=\frac9{11}\approx0.81818.$$

The predicted new value is about $100.81818$, while the exact value is $111/1.101\approx100.81744$. A changing rate offsets part of the payment increase. The approximation is not an exact finite-change identity. A shock of $0.1$ would be a hundred times larger than $0.001$.

A quadratic model has an exact remainder

For

$$V(x,y)=4x^2+2xy+9y^2,$$

we have $\nabla V=(8x+2y,2x+18y)^T$. If $h=(u,v)^T$, direct expansion gives

$$V(x+u,y+v)-V(x,y)=\nabla V(x,y)\cdot h+4u^2+2uv+9v^2.$$

Because $|2uv|\le u^2+v^2$, the remainder magnitude is at most $5u^2+10v^2\le10\|h\|^2$. Thus $\|h\|\le0.01$ guarantees an error at most $0.001$ in the output's units, using the chosen coordinate scales. A vanishing linear term does not force the quadratic change to vanish.

Follow a budget-preserving path

Let $C(x,y)=x^2+3y^2$ with $x+y=10$. Through $(4,6)$ use $x(t)=4+t$, $y(t)=6-t$. The chain rule gives

$$\frac{d}{dt}C(x(t),y(t))\bigg|_{t=0} =(2x,6y)\cdot(1,-1)\big|_{(4,6)}=8-36=-28.$$

Small positive $t$ decreases the cost to first order along this specified feasible path. This calculation does not by itself locate a global constrained optimum.

Practice with full solutions

P01. For $x=(1,-2,2)^T$ and $y=(3,0,1)^T$, find $x\cdot y$ and $\|x\|$.

Solution. $x\cdot y=3+0+2=5$ and $\|x\|=\sqrt{1+4+4}=3$.

P02. Find the gradient of $f(x,y)=x^2y+\ln y$ and state its real domain.

Solution. The domain is $\mathbb R\times(0,\infty)$. The gradient is $(2xy,x^2+1/y)^T$. The logarithm requires $y>0$; differentiation does not remove this restriction.

P03. For $P(F,r)=F/(1+r)$ at $(120,0.2)$, calculate $P$, $P_F$, and $P_r$.

Solution. $P=100$, $P_F=1/1.2=5/6$, and $P_r=-120/(1.2)^2=-250/3$.

P04. Estimate the change in P03 when $\Delta F=2$ and $\Delta r=0.002$. Interpret the rate unit.

Solution. The estimate is $(5/6)(2)-(250/3)(0.002)=5/3-1/6=3/2$. The rate rises by $0.002$, or $0.2$ percentage points, not by $0.2\%$ of its existing value.

P05. Differentiate $f(x(t),y(t))$ for $f(x,y)=xy+y^2$, $x(t)=t^2$, and $y(t)=1+t$. Verify by expansion.

Solution. $\nabla f=(y,x+2y)^T$, $x'=2t$, and $y'=1$. The derivative is $2t(1+t)+t^2+2(1+t)=3t^2+4t+2$. Expanding gives $f=t^3+2t^2+2t+1$, whose derivative agrees.

P06. Find the Jacobian of $g(x,y)=(x+y,xy)^T$. Use it to approximate $g(1.02,1.97)$ from $(1,2)$ and compare with the exact value.

Solution. $Dg=\begin{pmatrix}1&1\\y&x\end{pmatrix}$, so $Dg(1,2)=\begin{pmatrix}1&1\\2&1\end{pmatrix}$. For $h=(0.02,-0.03)^T$, the predicted change is $(-0.01,0.01)^T$, giving $(2.99,2.01)^T$. The exact second output is $1.02(1.97)=2.0094$, so exact minus predicted is $-0.0006$; the first output is exact.

P07. At $(1,2)$, find the directional derivative of $f=x^2+2y^2$ in the unit direction $(3/5,4/5)^T$.

Solution. The gradient is $(2x,4y)^T$, hence $(2,8)^T$ at the point. Its dot product with the given unit vector is $6/5+32/5=38/5$.

P08. For $Q(x,y)=x^2+2xy+4y^2$, find the exact first-order remainder and bound it by $5\|h\|^2$.

Solution. For $h=(u,v)^T$, the remainder is $u^2+2uv+4v^2$. Its magnitude is at most $u^2+2|uv|+4v^2\le2u^2+5v^2\le5(u^2+v^2)=5\|h\|^2$. Dividing by $\|h\|$ makes the remainder ratio tend to zero.

P09. Refute “both partial derivatives exist at a point, so the linearization is valid” using $f(0,0)=0$ and $f(x,y)=xy/(x^2+y^2)$ elsewhere. Compute both axis difference quotients and a diagonal limit.

Solution. Each coordinate restriction is zero, so $[f(h,0)-0]/h=0$ and $[f(0,h)-0]/h=0$ for nonzero $h$. Both partial derivatives are zero. But $f(t,t)=t^2/(2t^2)=1/2$ for $t\ne0$, not tending to the assigned value zero. The function is not continuous and therefore not differentiable at zero.

P10. For $g(x,y)=(x+y,xy)^T$ and $H(u,v)=u^2+v$, calculate $D(H\circ g)$ by Jacobian multiplication and verify by expansion.

Solution. The outer derivative is $DH(u,v)=(2u,1)$. Therefore

$$D(H\circ g)=(2(x+y),1)\begin{pmatrix}1&1\\y&x\end{pmatrix}=(2x+3y,3x+2y).$$

The expanded function is $(x+y)^2+xy=x^2+3xy+y^2$, whose two partial derivatives agree. The outer derivative is evaluated at $g(x,y)$ and multiplies the inner Jacobian on the left.