First-Order Expansion of the Recentred N-Agent Cost about a Stationary Mean-Field Triple and Its Coercive Lower Bound

Adopt the setting of the fluctuation processes of the controlled $N$-agent dynamics: a transition-rate family $\beta$ on $l$ states with control set $\mathcal{A}$, a nonempty subset of Euclidean space $\mathbb{R}^m$, and rate bound $B$; an observation-rate family $\tilde{\beta}$; a horizon $T>0$; an $N$-agent driving system (its probability measure is given no letter here, $P$ being reserved for the stationary co-state below); an observation-driven control policy $h$ which is $\mathcal{A}$-valued; a solution on $[0,T]$ with regular event $\Omega_0$, empirical state measure $\Sigma_t$, and control $\alpha_t$; a mean-field trajectory pair $(S,A)$ for $\beta$ with horizon $T$; and the fluctuation processes $\mathfrak{s}_t=\sqrt{N}(\Sigma_t-S_t)$ and $\mathfrak{a}_t=\sqrt{N}(\alpha_t-A_t)$. Let $(L,G)$ be population cost data on $l$ states with control dimension $m$, with $N$-agent cost $J^N[h]$ and mean-field cost $J^{MF}=J^{MF}[(S),(A)]$. Let $(U,V,\bar{\beta})$ be a twice continuously differentiable extension of $\beta$ with derivative bound $K$ and extended aggregate state drift $\bar{b}$ (the extended aggregate state drift), let $(U_c,\bar{L},\bar{G})$ be a twice continuously differentiable extension of $(L,G)$ with second-derivative bound $K_c$ — its open set, written $U$ in that definition and $W$ in the stationary-triple definition cited below, is written $U_c$ here as in the completion-of-squares theorem — and let $P$ be a stationary co-state for these data, so that $(S,A,P)$ is a stationary mean-field triple. Let $M=(M^1,\dots,M^l)$ be the martingale part of the martingale decomposition of the empirical state measure, so that

$$M^\gamma_t=\Sigma^\gamma_t-\Sigma^\gamma_0-\int_{[0,t]}\mathbf{1}_{\Omega_0}\,b^\gamma(\Sigma_s,\alpha_s)\,ds$$

at every point of $\Omega$, where $b$ is the aggregate state drift of $\beta$ and $\mathbf{1}_{D}$ denotes the function equal to $1$ on a set $D$ and $0$ off it. Adopt the partial-derivative notation $\partial_j\partial_i$ of the extension definitions, the mean-field Hamiltonian $\mathcal{H}_t(\Sigma,\alpha)=\bar{L}(\Sigma,\alpha)-\sum_{\delta=1}^{l}P^\delta_t\,\bar{b}^\delta(\Sigma,\alpha)$ and its state derivative coefficients $\partial_\gamma\mathcal{H}_t(\Sigma,\alpha)=\partial_\gamma\bar{L}(\Sigma,\alpha)-\sum_{\delta=1}^{l}P^\delta_t\,\partial_\gamma\bar{b}^\delta(\Sigma,\alpha)$ of the first-order expansion lemma for the mean-field cost, and the constants $M_2=K_c+3\,l\,K\,C_P$, $C_1=l\sqrt{l+m}\,M_2$ and $C_2=\tfrac{1}{2}(l+m)M_2$ of part (b) of that lemma, where $C_P$ is a fixed real number with $\sum_{\delta=1}^{l}|P^\delta_t|\le C_P$ for all $t\in[0,T]$ (existing by clause 1 of the co-state definition and the extreme value theorem). Write $|\cdot|$ for the Euclidean norm (Euclidean distance to the origin), $\mathbb{E}$ for the expectation, $\Delta^l$ for the probability simplex, and $\int_{[0,t]}\cdot\,ds$ for the Lebesgue integral over the compact interval $[0,t]$, taken to be $0$ for $t=0$. Throughout, a real-valued function on a subinterval $I$ of the real numbers $\mathbb{R}$ is called continuous on $I$ when it is continuous relative to $I$, both $I$ and the codomain $\mathbb{R}$ carrying the metric of the real line.

Hypothesis (A). The control set $\mathcal{A}$ is compact for the topology determined by the Euclidean distance and convex (compactness is used for the boundedness constants below, and compactness with convexity for part (b) of the first-order expansion lemma cited in the conclusions). In conclusions (d) and (e), hypotheses (H1) and (U) of the quadratic growth lemma for the mean-field Hamiltonian are additionally assumed, with $r_0>0$ the real number furnished by conclusion (d) of that lemma under (A), (H1), (U).

Fix, by claim 1 of the boundedness lemma for cost data over a compact control set, a real $C_{LG}\ge0$ with $|L(\Sigma,a)|\le C_{LG}$ and $|G(\Sigma)|\le C_{LG}$ for all $\Sigma\in\Delta^l$ and $a\in\mathcal{A}$; and fix a real $C_\partial\ge0$ with $\sum_{\gamma=1}^{l}\bigl|\partial_\gamma\mathcal{H}_t(S_t,A_t)\bigr|\le C_\partial$ for all $t\in[0,T]$, which exists because each map $t\mapsto\partial_\gamma\mathcal{H}_t(S_t,A_t)$ is continuous on $[0,T]$ (a finite sum of products of the continuous maps $t\mapsto P^\delta_t$ and of first-order partial derivatives of $\bar{L}$ and $\bar{b}^\delta$, continuous by the extension definitions and the regularity of the extended aggregate state drift, composed with the continuous $t\mapsto(S_t,A_t)$), hence bounded by the extreme value theorem. Define, for $t\in[0,T]$ and $\omega\in\Omega$, writing $y_t=\Sigma_t-S_t$ (so that $\mathfrak{s}_t=\sqrt{N}y_t$),

$$\mathcal{D}_t=\mathcal{H}_t(\Sigma_t,\alpha_t)-\mathcal{H}_t(S_t,A_t)-\sum_{\gamma=1}^{l}\partial_\gamma\mathcal{H}_t(S_t,A_t)\,y^\gamma_t,\qquad \mathcal{D}_G=\bar{G}(\Sigma_T)-\bar{G}(S_T)-\sum_{\gamma=1}^{l}\partial_\gamma\bar{G}(S_T)\,y^\gamma_T,$$

and set

$$\zeta_N=N\,\bigl(\mathbb{E}[\Sigma_0]-S_0\bigr)\in\mathbb{R}^l\ \text{(componentwise expectation)},\qquad \mathcal{J}_N=N\bigl(J^N[h]-J^{MF}\bigr)+\sum_{\gamma=1}^{l}P^\gamma_0\,\zeta^\gamma_N ,$$

the latter a well-defined real number because $J^N[h]$ is finite by conclusion (b) below.

Then the following hold.

(a) (Measurability and bounds.) The maps $(t,\omega)\mapsto\mathbf{1}_{\Omega_0}(\omega)\mathcal{D}_t(\omega)$ on $[0,T]\times\Omega$ and $(t,\omega)\mapsto\mathbf{1}_{\Omega_0}(\omega)L(\Sigma_t(\omega),\alpha_t(\omega))$ are measurable with respect to the product $\sigma$-algebra of the trace Borel $\sigma$-algebra on $[0,T]$ and $\mathcal{F}$, and $\mathcal{D}_G$ is a random variable. At every point of $[0,T]\times\Omega$ at which $\Sigma_t\in\Delta^l$ and $\alpha_t\in\mathcal{A}$,

$$|\mathcal{D}_t|\le C_{\mathcal{D}}=2\,C_{LG}+4(l-1)B\,C_P+C_\partial\,,$$

and at every point of $\Omega$ at which $\Sigma_T\in\Delta^l$, $|\mathcal{D}_G|\le C_{\mathcal{D}G}=2\,C_{LG}+C_P$; both memberships hold at every point of $[0,T]\times\Omega$, respectively $\Omega$, by the solution definition.

(b) (Pathwise first-order identity.) $J^N[h]$ is finite, and for every $\omega\in\Omega_0$ all integrals below exist and

$$\int_{[0,T]}L(\Sigma_t,\alpha_t)\,dt+G(\Sigma_T)-J^{MF}+\sum_{\gamma=1}^{l}P^\gamma_0\,y^\gamma_0=\int_{[0,T]}\mathcal{D}_t\,dt+\mathcal{D}_G+R^M,$$

where

$$R^M=-\sum_{\gamma=1}^{l}P^\gamma_T\,M^\gamma_T+\int_{[0,T]}\sum_{\gamma=1}^{l}\partial_\gamma\mathcal{H}_t(S_t,A_t)\,M^\gamma_t\,dt ,$$

and $R^M$ is taken to be $0$ at every $\omega\notin\Omega_0$, so that it is defined on all of $\Omega$.

(c) (Expectation identity.) The map $t\mapsto\mathbb{E}[\mathbf{1}_{\Omega_0}\mathcal{D}_t]$ is measurable and bounded on $[0,T]$, $\mathbb{E}[\mathbf{1}_{\Omega_0}R^M]=0$, and

$$\mathcal{J}_N=\int_{[0,T]}\mathbb{E}\bigl[\mathbf{1}_{\Omega_0}\,N\,\mathcal{D}_t\bigr]\,dt+\mathbb{E}\bigl[\mathbf{1}_{\Omega_0}\,N\,\mathcal{D}_G\bigr].$$

(d) (Coercive pointwise lower bound.) Under the additional hypotheses stated above, set $C_3=C_2+C_1^2/(2r_0)$. Then for every $\omega\in\Omega$ and every $t\in[0,T]$,

$$N\,\mathcal{D}_t\ \ge\ \frac{r_0}{2}\,|\mathfrak{a}_t|^2-C_3\,|\mathfrak{s}_t|^2 ,$$

and for every $\omega\in\Omega$,

$$N\,\mathcal{D}_G\ \ge\ -\frac{l\,K_c}{2}\,|\mathfrak{s}_T|^2 .$$

(e) (Energy inequality.) Under the same additional hypotheses, all terms below are finite and

$$\frac{r_0}{2}\int_{[0,T]}\mathbb{E}\bigl[\mathbf{1}_{\Omega_0}|\mathfrak{a}_t|^2\bigr]\,dt\ \le\ \mathcal{J}_N+C_3\int_{[0,T]}\mathbb{E}\bigl[\mathbf{1}_{\Omega_0}|\mathfrak{s}_t|^2\bigr]\,dt+\frac{l\,K_c}{2}\,\mathbb{E}\bigl[\mathbf{1}_{\Omega_0}|\mathfrak{s}_T|^2\bigr].$$