Proof of Multivariate Taylor Expansion with Uniform Second-Order Remainder

Throughout, the arithmetic and order of $\mathbb{R}$ are used through The Structure of ℝ as a Field and an Ordered Ring (i), The Structure of ℝ as a Field and an Ordered Ring (ii), Arithmetic and Order Rules in the Real Numbers, Stated for ℝ (v), Arithmetic and Order Rules in the Real Numbers, Stated for ℝ (ix), Rules of Arithmetic and Order in an Ordered Field (iii) and The Quotient 1/x, Order Rules for Sums, and Nesting of Closed Intervals in ℝ (i); norms and distances in $\mathbb{R}^{n}$ through Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (ii), Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (v), Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (vi) and Standard Basis Vectors of ℝ^q: Coordinates, Orthonormality and Unit Length (iii); the metric spaces $(\mathbb{R}^{n},d_{E})$ and $(\mathbb{R},d_{\mathbb{R}})$ through $\mathbb{R}^n$ with the Euclidean Distance Is a Complete Metric Space with Coordinatewise Convergence (i) and The Absolute Value Metric on the Real Numbers (i); the natural number $n$ and the numeral $2$ are read in $\mathbb{R}$ through Closure Facts in an Ordered Field: Negatives, Differences, Quotients, Absolute Values, Powers, Natural Numbers, Halves and Minima (v) and Closure Facts in an Ordered Field: Negatives, Differences, Quotients, Absolute Values, Powers, Natural Numbers, Halves and Minima (vi); values of composite maps such as $\varphi$ below through Basic Properties of Functions: Equality, Composition, Identity, Inverse and Restriction (ii); and the segment is written $[x,y]=\{x+th:\ t\in[0,1]\}$, since $(1-t)x+ty=x+t(y-x)$ (Segments and Convex Subsets of $\mathbb{R}^n$ (i)).

The hypotheses of (ii) and (iii) range over all pairs $(i,j)\in[n]\times[n]$, so they bound the Hessian entries $\partial_{i}\partial_{j}f(z)$ as well.

We first record a bound for quadratic forms, used for both (ii) and (iii). Let $B\in\mathcal{M}_{n}(\mathbb{R})$ and let $M\ge0$ be a real number with $|B_{ij}|\le M$ for all $i,j\in[n]$. Then

$$|h\cdot(Bh)|=\Bigl|\sum_{i=1}^{n}\sum_{j=1}^{n}h_{i}B_{ij}h_{j}\Bigr|\le M\Bigl(\sum_{i=1}^{n}|h_{i}|\Bigr)^{2}\le n\,M\,\lVert h\rVert^{2}.\tag{1}$$

The equality writes out the dot product and the matrix-vector product (The Dot Product on $\mathbb{R}^n$ (i), Matrix-Vector Product (i)) and moves each $h_{i}$ inside the inner sum, $h_{i}\sum_{j=1}^{n}B_{ij}h_{j}=\sum_{j=1}^{n}h_{i}B_{ij}h_{j}$, by distributivity for finite sums (Finite Sums in a Commutative Ring and in an Ordered Field: Distributivity, Differences, Telescoping, Constant Terms, Comparison and the Triangle Inequality (i)). The first inequality follows from the triangle inequality and comparison for finite sums (Finite Sums in a Commutative Ring and in an Ordered Field: Distributivity, Differences, Telescoping, Constant Terms, Comparison and the Triangle Inequality (vii), Finite Sums in a Commutative Ring and in an Ordered Field: Distributivity, Differences, Telescoping, Constant Terms, Comparison and the Triangle Inequality (v)), with distributivity used again to expand $\bigl(\sum_{i=1}^{n}|h_{i}|\bigr)^{2}=\sum_{i=1}^{n}\sum_{j=1}^{n}|h_{i}|\,|h_{j}|$. The last inequality comes from

$$\sum_{i=1}^{n}|h_{i}|\le\sqrt{n}\,\lVert h\rVert,$$

which follows from the Cauchy--Schwarz inequality The Cauchy-Schwarz Inequality on $\mathbb{R}^n$ (i) applied to $a=(1,\dots,1)$ and $b=(|h_{1}|,\dots,|h_{n}|)$:

$$\sum_{i=1}^{n}|h_{i}|=a\cdot b\le|a\cdot b|\le\lVert a\rVert\,\lVert b\rVert.$$

Here $\lVert b\rVert=\lVert h\rVert$, since $|h_{i}|^{2}=h_{i}^{2}$ for every $i$ and both norms are the square roots of the sums of these squares (Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (i)), and $\lVert a\rVert=\sqrt{n}$, since $\lVert a\rVert^{2}=n$ (Finite Sums in a Commutative Ring and in an Ordered Field: Distributivity, Differences, Telescoping, Constant Terms, Comparison and the Triangle Inequality (iv)).

We now prove (i). Since $f$ is of class $C^{1}$, it is differentiable on $U$ by Basic Properties of Cᵏ Functions (v). Let $J=[0,1]$ and $\varphi(t)=f(x+th)$ for $t\in J$, which is defined because $[x,y]\subseteq U$. By Derivatives of a Function Along a Line Segment in ℝⁿ (ii),

$$\varphi'(t)=\nabla f(x+th)\cdot h\qquad(t\in J),$$

and $\varphi$, being differentiable, is continuous on $J$ (The Increment Form of the Derivative, and Differentiability Implies Continuity (ii)). The mean value theorem Mean Value Theorem on a Closed Real Interval and on Any Interval (i) therefore gives $t\in(0,1)$ with $\varphi(1)-\varphi(0)=\varphi'(t)(1-0)$. Since $h=y-x$, we have $\varphi(0)=f(x)$ and $\varphi(1)=f(x+h)=f(y)$, so

$$f(y)-f(x)=\varphi(1)-\varphi(0)=\varphi'(t)(1-0)=\nabla f(z)\cdot h,\qquad z=x+th\in[x,y].$$

By the Cauchy--Schwarz inequality The Cauchy-Schwarz Inequality on $\mathbb{R}^n$ (i), $|\nabla f(z)\cdot h|\le\lVert\nabla f(z)\rVert\,\lVert h\rVert$. Since $z\in[x,y]$, every component $\partial_{i}f(z)$ of $\nabla f(z)$ satisfies $|\partial_{i}f(z)|\le M_{1}$, so

$$\lVert\nabla f(z)\rVert^{2}=\sum_{i=1}^{n}\partial_{i}f(z)^{2}\le nM_{1}^{2}$$

(Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (i), Gradient of a Real-Valued Function on an Open Subset of $\mathbb{R}^n$ (i), Finite Sums in a Commutative Ring and in an Ordered Field: Distributivity, Differences, Telescoping, Constant Terms, Comparison and the Triangle Inequality (v), Finite Sums in a Commutative Ring and in an Ordered Field: Distributivity, Differences, Telescoping, Constant Terms, Comparison and the Triangle Inequality (iv)). As both sides are nonnegative, taking square roots gives $\lVert\nabla f(z)\rVert\le\sqrt{n}\,M_{1}$ (The k-th Root and the Square Root of a Nonnegative Real Number (ii)), and hence

$$|f(y)-f(x)|\le\lVert\nabla f(z)\rVert\,\lVert h\rVert\le\sqrt{n}\,M_{1}\,\lVert h\rVert.$$

For (ii), let $f$ be of class $C^{2}$. By Second-Order Taylor Expansion (i) with $a=x$, there is $t\in(0,1)$ such that, with $z=x+th\in[x,y]$,

$$f(y)-f(x)-\nabla f(x)\cdot h=\tfrac{1}{2}\,h\cdot\bigl(D^{2}f(z)\,h\bigr).$$

The entries of $D^{2}f(z)$ are the values $\partial_{i}\partial_{j}f(z)$ (The Hessian Matrix of a Function of Class C² (i)), which are bounded in absolute value by $M_{2}$. So (1) with $B=D^{2}f(z)$ and $M=M_{2}$ gives

$$\bigl|f(y)-f(x)-\nabla f(x)\cdot h\bigr|\le\tfrac{1}{2}\,n\,M_{2}\,\lVert h\rVert^{2}.$$

For (iii), take $t$ and $z$ as in the previous paragraph, and write $B=D^{2}f(z)-D^{2}f(x)$, so that $D^{2}f(z)=B+D^{2}f(x)$. Distributivity of the matrix-vector product, $(A+A')v=Av+A'v$ (Algebraic Rules for Matrix Products, Matrix-Vector Products and Transposes (v)), gives $D^{2}f(z)h=Bh+D^{2}f(x)h$, that is,

$$Bh=D^{2}f(z)h-D^{2}f(x)h.$$

Bilinearity of the dot product (The Dot Product on $\mathbb{R}^n$ Is Symmetric, Bilinear and Positive Definite (ii), The Dot Product on $\mathbb{R}^n$ Is Symmetric, Bilinear and Positive Definite (iii)) then turns the identity of the previous paragraph into

$$f(y)-f(x)-\nabla f(x)\cdot h-\tfrac{1}{2}\,h\cdot\bigl(D^{2}f(x)\,h\bigr)=\tfrac{1}{2}\,h\cdot(Bh).$$

The entries of $B$ are $\partial_{i}\partial_{j}f(z)-\partial_{i}\partial_{j}f(x)$ (The Hessian Matrix of a Function of Class C² (i) and Real Matrices: the Set of Real m x n Matrices with Their Entrywise Operations (iv)), which are bounded in absolute value by $\bar{\varepsilon}$. So (1) with $M=\bar{\varepsilon}$ gives

$$\Bigl|f(y)-f(x)-\nabla f(x)\cdot h-\tfrac{1}{2}\,h\cdot\bigl(D^{2}f(x)\,h\bigr)\Bigr|\le\tfrac{1}{2}\,n\,\bar{\varepsilon}\,\lVert h\rVert^{2}.$$