Proof of Second-Order Taylor Expansion

Throughout, the arithmetic and order of $\mathbb{R}$ are used through The Structure of ℝ as a Field and an Ordered Ring (i), The Structure of ℝ as a Field and an Ordered Ring (ii), Arithmetic and Order Rules in the Real Numbers, Stated for ℝ (v), Arithmetic and Order Rules in the Real Numbers, Stated for ℝ (ix), Rules of Arithmetic and Order in an Ordered Field (iii) and The Quotient 1/x, Order Rules for Sums, and Nesting of Closed Intervals in ℝ (i); norms and distances in $\mathbb{R}^{n}$ through Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (ii), Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (v), Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (vi) and Standard Basis Vectors of ℝ^q: Coordinates, Orthonormality and Unit Length (iii); and the metric spaces $(\mathbb{R}^{n},d_{E})$ and $(\mathbb{R},d_{\mathbb{R}})$ through $\mathbb{R}^n$ with the Euclidean Distance Is a Complete Metric Space with Coordinatewise Convergence (i) and The Absolute Value Metric on the Real Numbers (i).

We begin with (i), which we reduce to one variable. Let $h\in\mathbb{R}^{n}$ with $[a,a+h]\subseteq U$, and let $J=[0,1]$, an interval containing two distinct points. For $t\in J$ we have $(1-t)a+t(a+h)=a+th$, so by the definition of the segment the hypothesis says that $a+th\in U$ for every $t\in J$. Let $\varphi:J\to\mathbb{R}$ be the function $\varphi(t)=f(a+th)$. Since $f$ is of class $C^{2}$ on $U$, it is of class $C^{1}$ (Basic Properties of Cᵏ Functions (ii)) and hence differentiable on $U$ (Basic Properties of Cᵏ Functions (v)). By Derivatives of a Function Along a Line Segment in ℝⁿ (ii) and Derivatives of a Function Along a Line Segment in ℝⁿ (iv), $\varphi$ is differentiable on $J$, its derivative $\varphi'$ is again differentiable on $J$, and for every $t\in J$

$$\varphi'(t)=\nabla f(a+th)\cdot h,\qquad \varphi''(t)=h\cdot\bigl(D^{2}f(a+th)\,h\bigr).$$

It remains to prove the one-variable Taylor formula of order two for $\varphi$ on $[0,1]$; we argue as in the proof of Taylor's theorem in one variable, through Rolle's theorem. We subtract from $\varphi$ its first-order Taylor polynomial at $0$ and a quadratic correction chosen so that the difference vanishes at both endpoints: let

$$c=\varphi(1)-\varphi(0)-\varphi'(0),\qquad g(t)=\varphi(t)-\varphi(0)-\varphi'(0)\,t-c\,t^{2}\quad(t\in J).$$

Then $g(0)=0$ and $g(1)=\varphi(1)-\varphi(0)-\varphi'(0)-c=0$. By the rules for derivatives of constants, of the identity, of sums, of scalar multiples and of powers (Derivatives of Constants, the Identity, Sums, Scalar Multiples, Products, Reciprocals, Quotients and Powers (i), Derivatives of Constants, the Identity, Sums, Scalar Multiples, Products, Reciprocals, Quotients and Powers (ii), Derivatives of Constants, the Identity, Sums, Scalar Multiples, Products, Reciprocals, Quotients and Powers (iv), Derivatives of Constants, the Identity, Sums, Scalar Multiples, Products, Reciprocals, Quotients and Powers (v), Derivatives of Constants, the Identity, Sums, Scalar Multiples, Products, Reciprocals, Quotients and Powers (iii)), $g$ is differentiable on $J$, $g'$ is differentiable on $J$, and

$$g'(t)=\varphi'(t)-\varphi'(0)-2ct,\qquad g''(t)=\varphi''(t)-2c\qquad(t\in J).$$

Being differentiable, $g$ and $g'$ are continuous on $J$ (The Increment Form of the Derivative, and Differentiability Implies Continuity (ii)).

Since $g(0)=g(1)$, Rolle's theorem (Rolle's Theorem on a Closed Real Interval (i)) gives $s\in(0,1)$ with $g'(s)=0$. Moreover $g'(0)=\varphi'(0)-\varphi'(0)-0=0$. The restriction of $g'$ to $[0,s]$ is continuous on $[0,s]$ and differentiable at every point of $[0,s]$, with derivative $g''$ there (Restriction Stability of Continuity and of the Derivative (i), Restriction Stability of Continuity and of the Derivative (ii)), and it vanishes at both endpoints. A second application of Rolle's Theorem on a Closed Real Interval (i), now to this restriction on $[0,s]$, gives $t\in(0,s)$ with $g''(t)=0$, that is, $\varphi''(t)=2c$. Then $t\in(0,1)$, and substituting $c=\tfrac{1}{2}\varphi''(t)$ into the definition of $c$ and the formulas for $\varphi'$ and $\varphi''$ gives

$$f(a+h)=\varphi(1)=\varphi(0)+\varphi'(0)+\tfrac{1}{2}\varphi''(t)=f(a)+\nabla f(a)\cdot h+\tfrac{1}{2}\,h\cdot\bigl(D^{2}f(a+th)\,h\bigr),$$

which is (i).

We turn to (ii). Let $\varepsilon>0$; we choose two radii, in this order. First, since $U$ is open and $a\in U$, there is $\rho>0$ with $B(a,\rho)\subseteq U$. Second, the Hessian function $D^{2}f:U\to\mathcal{M}_{n}(\mathbb{R})$ is continuous on $U$ by Equality of Mixed Partial Derivatives, and Symmetry and Continuity of the Hessian (iii), its values carrying the metric $d_{n}(A,B)=\lVert A-B\rVert$ of Euclidean Space and Real Matrices: Standing Notation (vi). Applying the definition of continuity at $a$ with the positive number $2\varepsilon$, and shrinking the radius it provides to at most $\rho$, we obtain $\delta$ with $0<\delta\le\rho$ such that every $z\in U$ with $\lVert z-a\rVert<\delta$ ($d_{E}(a,z)=\lVert z-a\rVert$ by Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (ii)) satisfies

$$\lVert D^{2}f(z)-D^{2}f(a)\rVert\le2\varepsilon .$$

Thus $\delta$ is fixed by $\varepsilon$ (for the given $f$ and $a$) before the increment $h$ is chosen.

Now let $h\in\mathbb{R}^{n}$ with $\lVert h\rVert<\delta$. For $t\in[0,1]$,

$$\lVert(a+th)-a\rVert=t\,\lVert h\rVert\le\lVert h\rVert<\delta$$

(Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (v)), so every point of $[a,a+h]$ lies in $B(a,\delta)$, which is contained in $B(a,\rho)\subseteq U$ since $\delta\le\rho$. In particular $a+h\in U$, and since $[a,a+h]\subseteq U$, (i) gives $t\in(0,1)$ with

$$f(a+h)=f(a)+\nabla f(a)\cdot h+\tfrac{1}{2}\,h\cdot\bigl(D^{2}f(a+th)\,h\bigr).$$

Let $A=D^{2}f(a+th)-D^{2}f(a)$. Subtracting $\nabla f(a)\cdot h+\tfrac{1}{2}\,h\cdot\bigl(D^{2}f(a)\,h\bigr)$ and using the linearity of the dot product in its second argument (The Dot Product on $\mathbb{R}^n$ Is Symmetric, Bilinear and Positive Definite (iii)) and the distributivity of the matrix-vector product, we find

$$f(a+h)-f(a)-\nabla f(a)\cdot h-\tfrac{1}{2}\,h\cdot\bigl(D^{2}f(a)\,h\bigr)=\tfrac{1}{2}\,h\cdot(Ah).$$

Both Hessian matrices lie in $\mathcal{S}(n)$ by Equality of Mixed Partial Derivatives, and Symmetry and Continuity of the Hessian (ii), and $\mathcal{S}(n)$ is closed under differences (Rules for Symmetric and Positive Semidefinite Matrices: Closure, Self-Adjointness, Polarization, Congruence and Cones (i)), so $A\in\mathcal{S}(n)$ and The Operator Norm of a Symmetric Matrix: Quadratic Forms, Eigenvalues and the Semidefinite Ordering (iii) gives $|h\cdot(Ah)|\le\lVert A\rVert\,\lVert h\rVert^{2}$. Finally, the point $z=a+th$ lies in $U$ and satisfies $\lVert z-a\rVert<\delta$ by the estimate above, so the choice of $\delta$ gives $\lVert A\rVert\le2\varepsilon$. Hence

$$\Bigl|\,f(a+h)-f(a)-\nabla f(a)\cdot h-\tfrac{1}{2}\,h\cdot\bigl(D^{2}f(a)\,h\bigr)\Bigr|=\tfrac{1}{2}\,|h\cdot(Ah)|\le\tfrac{1}{2}\lVert A\rVert\,\lVert h\rVert^{2}\le\varepsilon\lVert h\rVert^{2},$$

which proves (ii).