Proof of Chain Rule for Differentiable Functions Between Euclidean Spaces

Throughout, the arithmetic and order of $\mathbb{R}$ are used through The Structure of ℝ as a Field and an Ordered Ring (i), The Structure of ℝ as a Field and an Ordered Ring (ii), Arithmetic and Order Rules in the Real Numbers, Stated for ℝ (v), Arithmetic and Order Rules in the Real Numbers, Stated for ℝ (ix), Rules of Arithmetic and Order in an Ordered Field (iii) and The Quotient 1/x, Order Rules for Sums, and Nesting of Closed Intervals in ℝ (i); norms and distances in $\mathbb{R}^{n}$ through Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (ii), Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (v), Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (vi) and Standard Basis Vectors of ℝ^q: Coordinates, Orthonormality and Unit Length (iii); and the metric spaces $(\mathbb{R}^{n},d_{E})$ and $(\mathbb{R},d_{\mathbb{R}})$ through $\mathbb{R}^n$ with the Euclidean Distance Is a Complete Metric Space with Coordinatewise Convergence (i) and The Absolute Value Metric on the Real Numbers (i). By The Derivative Is the Jacobian Matrix, and Is Unique (ii) and The Derivative Is the Jacobian Matrix, and Is Unique (i), a function $\varphi$ differentiable at a point $x$ is differentiable there with derivative matrix $D\varphi(x)$, or with derivative vector $\nabla\varphi(x)$ when it is real-valued, and with no other. So in (i) and (ii) it suffices to exhibit a derivative matrix or a derivative vector of $g\circ f$ at $a$, in the sense of Differentiability at a Point for Functions Between Euclidean Spaces (i) or Differentiability at a Point for Functions Between Euclidean Spaces (ii); it is then $D(g\circ f)(a)$ or $\nabla(g\circ f)(a)$.

Step 1 ((i)). Write $b=f(a)$, $A=Df(a)$ and $B=Dg(b)$, so that $f$ is differentiable at $a$ with derivative matrix $A$ and $g$ is differentiable at $b$ with derivative matrix $B$. For $h\in\mathbb{R}^{n}$ with $a+h\in U$ put

$$k(h)=f(a+h)-f(a),\qquad u(h)=f(a+h)-f(a)-Ah,$$

and for $\kappa\in\mathbb{R}^{m}$ with $b+\kappa\in V$ put $v(\kappa)=g(b+\kappa)-g(b)-B\kappa$. Since $b+k(h)=f(a+h)\in V$, the vector $v(k(h))$ is defined, and since $k(h)=Ah+u(h)$, the rules $(BA)h=B(Ah)$ and $B(x+y)=Bx+By$ of Algebraic Rules for Matrix Products, Matrix-Vector Products and Transposes (v) give

$$g\bigl(f(a+h)\bigr)-g\bigl(f(a)\bigr)-(BA)h=v\bigl(k(h)\bigr)+B\,u(h).$$

The remainder of $g\circ f$ thus splits into the remainder of $g$ at the increment $k(h)$ of $f$, which is small compared with $\lVert k(h)\rVert$ and hence with $\lVert h\rVert$, and the image under $B$ of the remainder of $f$. Let $\varepsilon>0$. We choose, in this order: the tolerance $\eta=\varepsilon/(\lVert A\rVert+\lVert B\rVert+1)>0$; a $\delta_{g}>0$ given by Differentiability at a Point for Functions Between Euclidean Spaces (i) for $g$ at $b$ with tolerance $\eta$, so that every $\kappa$ with $\lVert\kappa\rVert<\delta_{g}$ satisfies $b+\kappa\in V$ and $\lVert v(\kappa)\rVert\le\eta\lVert\kappa\rVert$; a $\delta_{0}>0$ given by Differentiable Functions Are Continuous (ii) for $f$ at $a$, so that every $h$ with $\lVert h\rVert<\delta_{0}$ satisfies $a+h\in U$ and

$$\lVert k(h)\rVert\le\bigl(\lVert A\rVert+1\bigr)\lVert h\rVert;$$

and a $\delta_{f}>0$ given by Differentiability at a Point for Functions Between Euclidean Spaces (i) for $f$ at $a$ with tolerance $\eta$, so that every $h$ with $\lVert h\rVert<\delta_{f}$ satisfies $\lVert u(h)\rVert\le\eta\lVert h\rVert$. Set

$$\delta=\min\Bigl(\delta_{0},\delta_{f},\frac{\delta_{g}}{\lVert A\rVert+1}\Bigr).$$

Let $\lVert h\rVert<\delta$. Then $a+h\in U$ and $\lVert k(h)\rVert\le(\lVert A\rVert+1)\lVert h\rVert<\delta_{g}$, so $\lVert v(k(h))\rVert\le\eta\lVert k(h)\rVert\le\eta(\lVert A\rVert+1)\lVert h\rVert$, while $\lVert B\,u(h)\rVert\le\lVert B\rVert\,\lVert u(h)\rVert\le\lVert B\rVert\,\eta\lVert h\rVert$ by Rules for the Operator Norm: Bound, Least Constant, Definiteness, Homogeneity, Triangle Inequality and Products (ii). The identity above and Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (vi) give

$$\bigl\lVert g\bigl(f(a+h)\bigr)-g\bigl(f(a)\bigr)-(BA)h\bigr\rVert\le\eta\bigl(\lVert A\rVert+1\bigr)\lVert h\rVert+\eta\lVert B\rVert\,\lVert h\rVert=\varepsilon\lVert h\rVert .$$

So $g\circ f$ is differentiable at $a$ with derivative matrix $BA$, and $D(g\circ f)(a)=BA=Dg(f(a))\,Df(a)$.

Step 2 ((ii)). Now $g$ is real-valued. Keep $b$, $A$, $k(h)$ and $u(h)$ from Step 1, let $w=\nabla g(b)$, and for $b+\kappa\in V$ put $v(\kappa)=g(b+\kappa)-g(b)-w\cdot\kappa$. The identity $w\cdot(Ah)=(A^{\top}w)\cdot h$ (Algebraic Rules for Matrix Products, Matrix-Vector Products and Transposes (viii) with The Dot Product on $\mathbb{R}^n$ Is Symmetric, Bilinear and Positive Definite (i)) and the linearity $w\cdot(Ah+u)=w\cdot Ah+w\cdot u$ (The Dot Product on $\mathbb{R}^n$ Is Symmetric, Bilinear and Positive Definite (iii)) give

$$g\bigl(f(a+h)\bigr)-g\bigl(f(a)\bigr)-(A^{\top}w)\cdot h=v\bigl(k(h)\bigr)+w\cdot u(h),$$

and the Cauchy--Schwarz inequality The Cauchy-Schwarz Inequality on $\mathbb{R}^n$ (i) bounds the last term by

$$|w\cdot u(h)|\le\lVert w\rVert\,\lVert u(h)\rVert .$$

Given $\varepsilon>0$, the choices of Step 1 are made in the same order, with $\lVert w\rVert$ in place of $\lVert B\rVert$, so $\eta=\varepsilon/(\lVert A\rVert+\lVert w\rVert+1)$, and with $\delta_{g}$ now given by Differentiability at a Point for Functions Between Euclidean Spaces (ii) for $g$ at $b$ with tolerance $\eta$. The same estimate yields $|g(f(a+h))-g(f(a))-(A^{\top}w)\cdot h|\le\varepsilon\lVert h\rVert$ whenever $\lVert h\rVert<\delta$. So $A^{\top}w$ is a derivative vector of $g\circ f$ at $a$, and $\nabla(g\circ f)(a)=Df(a)^{\top}\nabla g(f(a))$.

Step 3 ((iii)). Write $x=\gamma(t)$, $c=\gamma'(t)$ and $A=Df(x)$. For $s\in I$ with $s\neq t$ put

$$h(s)=\gamma(s)-\gamma(t),\qquad z(s)=\frac{h(s)}{s-t}-c,$$

so that $\gamma(s)=x+h(s)$, and by Algebraic Rules for Matrix Products, Matrix-Vector Products and Transposes (v)

$$\frac{f(\gamma(s))-f(\gamma(t))}{s-t}-Ac=\frac{f(x+h(s))-f(x)-Ah(s)}{s-t}+A\,z(s).$$

The first term is the remainder of $f$ at $x$ divided by $s-t$; the second is small because the difference quotients of $\gamma$ tend to $c$. Let $\varepsilon>0$. We choose, in this order: the tolerance $\eta=\varepsilon/(2(\lVert c\rVert+1))$; a $\delta_{f}>0$ given by Differentiability at a Point for Functions Between Euclidean Spaces (i) for $f$ at $x$ with tolerance $\eta$; a $\delta_{1}>0$ given by The Derivative of a Vector-Valued Function as a Limit of Difference Quotients in Norm (i) for $\gamma$ at $t$ with tolerance $\min\bigl(1,\varepsilon/(2(\lVert A\rVert+1))\bigr)$, so that $\lVert z(s)\rVert$ is below this tolerance whenever $s\in I$ and $0<|s-t|<\delta_{1}$; and

$$\delta=\min\Bigl(\delta_{1},\frac{\delta_{f}}{\lVert c\rVert+1}\Bigr).$$

Let $s\in I$ with $0<|s-t|<\delta$. Then $\lVert z(s)\rVert<1$, and since $h(s)=(s-t)\bigl(c+z(s)\bigr)$, Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (vi) and Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (v) give

$$\lVert h(s)\rVert\le\bigl(\lVert c\rVert+1\bigr)|s-t|<\delta_{f}.$$

Hence $\lVert f(x+h(s))-f(x)-Ah(s)\rVert\le\eta\lVert h(s)\rVert\le\eta(\lVert c\rVert+1)|s-t|$, and the first term has norm at most $\eta(\lVert c\rVert+1)=\varepsilon/2$. By Rules for the Operator Norm: Bound, Least Constant, Definiteness, Homogeneity, Triangle Inequality and Products (ii) the second term has norm at most $\lVert A\rVert\,\lVert z(s)\rVert\le\lVert A\rVert\,\varepsilon/(2(\lVert A\rVert+1))<\varepsilon/2$. So

$$\Bigl\lVert\frac{f(\gamma(s))-f(\gamma(t))}{s-t}-Ac\Bigr\rVert<\varepsilon ,$$

and The Derivative of a Vector-Valued Function as a Limit of Difference Quotients in Norm (i), applied to $f\circ\gamma$ and $Ac$, shows that $f\circ\gamma$ is differentiable at $t$ with $(f\circ\gamma)'(t)=Ac=Df(\gamma(t))\,\gamma'(t)$.

Step 4 ((iv)). Now $f$ is real-valued. Keep $x$, $c$, $h(s)$ and $z(s)$ from Step 3 and let $w=\nabla f(x)$. Since $w\cdot\frac{h(s)}{s-t}-w\cdot c=w\cdot z(s)$ by The Dot Product on $\mathbb{R}^n$ Is Symmetric, Bilinear and Positive Definite (iii),

$$\frac{f(\gamma(s))-f(\gamma(t))}{s-t}-w\cdot c=\frac{f(x+h(s))-f(x)-w\cdot h(s)}{s-t}+w\cdot z(s),$$

and $|w\cdot z(s)|\le\lVert w\rVert\,\lVert z(s)\rVert$ by The Cauchy-Schwarz Inequality on $\mathbb{R}^n$ (i). Given $\varepsilon>0$, the choices of Step 3 are made in the same order, with $\lVert w\rVert$ in place of $\lVert A\rVert$ and with $\delta_{f}$ now given by the remainder estimate of Differentiability at a Point for Functions Between Euclidean Spaces (ii) for $f$ at $x$ with tolerance $\eta$. As in Step 3 the first term has absolute value at most $\varepsilon/2$, and by The Cauchy-Schwarz Inequality on $\mathbb{R}^n$ (i) the second has absolute value at most $\lVert w\rVert\,\lVert z(s)\rVert\le\lVert w\rVert\,\varepsilon/(2(\lVert w\rVert+1))<\varepsilon/2$. So every $s\in I$ with $0<|s-t|<\delta$ satisfies

$$\Bigl|\frac{f(\gamma(s))-f(\gamma(t))}{s-t}-w\cdot c\Bigr|<\varepsilon .$$

This is the condition of Derivative of a Real Function on an Interval (i) itself: $w\cdot c$ is a derivative of $f\circ\gamma$ at $t$, so $f\circ\gamma$ is differentiable at $t$ and, the derivative being unique (Derivative of a Real Function on an Interval (ii)), $(f\circ\gamma)'(t)=\nabla f(\gamma(t))\cdot\gamma'(t)$.