Second-Order Expansion of the N-Agent Cost about a Stationary Mean-Field Trajectory

Adopt the setting of the fluctuation processes of the controlled $N$-agent dynamics: a transition-rate family $\beta$ on $l$ states with control set $\mathcal{A}$, a nonempty subset of Euclidean space $\mathbb{R}^m$, and rate bound $B$, an observation-rate family $\tilde{\beta}$, a horizon $T>0$, an $N$-agent driving system, an observation-driven control policy $h$ which is $\mathcal{A}$-valued, a solution on $[0,T]$ with regular event $\Omega_0$, empirical state measure $\Sigma_t$, and control $\alpha_t$, a mean-field trajectory pair $(S,A)$ for $\beta$ with horizon $T$, and the fluctuation processes $\mathfrak{s}_t=\sqrt{N}(\Sigma_t-S_t)$ and $\mathfrak{a}_t=\sqrt{N}(\alpha_t-A_t)$. Let $(L,G)$ be population cost data on $l$ states with control dimension $m$, with $N$-agent cost $J^N[h]$ and mean-field cost $J^{MF}[(S),(A)]$. Let $(U,V,\bar{\beta})$ be a twice continuously differentiable extension of $\beta$ with derivative bound $K$, let $\bar{b}$ be the extended aggregate state drift of $(U,V,\bar{\beta})$, let $(W,\bar{L},\bar{G})$ be a twice continuously differentiable extension of $(L,G)$ with second-derivative bound $K_c$, and let $P$ be a stationary co-state for these data, so that $(S,A,P)$ is a stationary mean-field triple. Adopt the partial-derivative notation $\partial_j\partial_i$ of the extension definitions, write $d$ for the Euclidean distance, $|\cdot|$ for the Euclidean norm, $\mathbb{E}$ for the expectation, and $\Delta^l$ for the probability simplex, and let $C_P$ be a real number with $\sum_{\delta=1}^{l}|P^\delta_t|\le C_P$ for all $t\in[0,T]$, which exists because each component of $P$ is continuous on $[0,T]$ by clause 1 of the co-state definition - relative to $[0,T]$, with the metric of the real line - and therefore attains a maximum and a minimum on $[0,T]$ by the extreme value theorem, hence is bounded.

Hypotheses. Assume that the control set $\mathcal{A}$ is convex and that $\mathcal{A}_2=\int_{[0,T]}\mathbb{E}[|\mathfrak{a}_t|^2]\,dt<\infty$, this integral being well defined by part (a) of the a priori second-moment bound; as in that lemma, the subscripted symbol $\mathcal{A}_2$ is distinct from the control set $\mathcal{A}$.

For $u\ge0$ define

$$\omega_L(u)=\sup\big\{|\partial_j\partial_i\bar{L}(x)-\partial_j\partial_i\bar{L}(y)|:\ i,j\in\{1,\dots,l+m\},\ x,y\in\Delta^l\times\mathbb{R}^m,\ d(x,y)\le u\big\},$$ $$\omega_b(u)=\sup\big\{|\partial_j\partial_i\bar{b}^\gamma(x)-\partial_j\partial_i\bar{b}^\gamma(y)|:\ \gamma\in\{1,\dots,l\},\ i,j\in\{1,\dots,l+m\},\ x,y\in\Delta^l\times V,\ d(x,y)\le u\big\},$$ $$\omega_G(u)=\sup\big\{|\partial_\delta\partial_\gamma\bar{G}(\Sigma)-\partial_\delta\partial_\gamma\bar{G}(\Sigma')|:\ \gamma,\delta\in\{1,\dots,l\},\ \Sigma,\Sigma'\in\Delta^l,\ d(\Sigma,\Sigma')\le u\big\}.$$

(a) (Moduli.) $\omega_L$, $\omega_b$, and $\omega_G$ are nondecreasing functions from $[0,\infty)$ to $[0,\infty)$ with $\omega_L\le2K_c$, $\omega_b\le6\,l\,K$, and $\omega_G\le2K_c$ everywhere, and for every $\varepsilon>0$ there is $\delta>0$ such that $\omega_L(u)\le\varepsilon$, $\omega_b(u)\le\varepsilon$, and $\omega_G(u)\le\varepsilon$ for all $u\in[0,\delta]$.

(b) (Finiteness and eligibility.) $J^N[h]$ is finite; the pair $((\mathfrak{s}),(\mathfrak{a}))$ satisfies requirements (i)-(iii) of the fluctuation linear-quadratic cost (requirement (i) holding with $\Omega_1=\Omega_0$ by the joint measurability of the state and control), so that $LQG[(\mathfrak{s}),(\mathfrak{a})]$ is a well-defined real number; and all expectations and integrals appearing in (c) are well defined and finite.

(c) (Expansion with quantitative remainder.) Set $\zeta_N=N\,(\mathbb{E}[\Sigma_0]-S_0)\in\mathbb{R}^l$, with the componentwise expectation, and set $\rho_t=d\big((\Sigma_t,\alpha_t),(S_t,A_t)\big)$ for $t\in[0,T]$, so that $\rho_t=N^{-1/2}\big(|\mathfrak{s}_t|^2+|\mathfrak{a}_t|^2\big)^{1/2}$ and $d(\Sigma_T,S_T)=N^{-1/2}|\mathfrak{s}_T|$. Then the real number $R_N$ defined by the identity

$$N\Big(J^N[h]-J^{MF}[(S),(A)]\Big)=LQG\big[(\mathfrak{s}),(\mathfrak{a})\big]-\sum_{\gamma=1}^{l}P^\gamma_0\,\zeta^\gamma_N+R_N$$

satisfies

$$|R_N|\ \le\ \frac{l+m}{2}\int_{[0,T]}\mathbb{E}\Big[\big(\omega_L(\rho_t)+C_P\,\omega_b(\rho_t)\big)\big(|\mathfrak{s}_t|^2+|\mathfrak{a}_t|^2\big)\Big]\,dt\ +\ \frac{l}{2}\,\mathbb{E}\Big[\omega_G\big(d(\Sigma_T,S_T)\big)\,|\mathfrak{s}_T|^2\Big].$$