Proof of First-Order Expansion of the Recentred N-Agent Cost about a Stationary Mean-Field Triple and Its Coercive Lower Bound

Throughout, "the first-order lemma" is the first-order expansion lemma for the mean-field cost, "the growth lemma" is the quadratic growth lemma, "the toolkit" is the integral toolkit on a compact interval, "the covariation lemma" is the stopped covariation lemma, and "the existence theorem" is the existence and uniqueness theorem. All notation is that of the statement. We use repeatedly: $\Sigma_t(\omega)\in\Delta^l$ at every point of $[0,T]\times\Omega$ (clause (vii)(a) of the existence theorem), and $\alpha_t(\omega)\in\mathcal{A}$ at every point of $[0,T]\times\Omega$, the control process of a solution being $\mathcal{A}$-valued by that definition. In particular $\mathcal{D}_t$ is defined at every point of $[0,T]\times\Omega$. Moreover $\Delta^l\subseteq U\cap U_c$ and $\mathcal{A}\subseteq V$ by the transition-rate and cost extension definitions, $\bar{L}=L$ on $\Delta^l\times\mathbb{R}^m$ and $\bar{G}=G$ on $\Delta^l$ by the extension property of the cost extension definition, and $\bar{b}^\delta=b^\delta$ on $\Delta^l\times\mathcal{A}$ by part (i) of the regularity of the extended aggregate state drift. Finally, for $(\Sigma,a)\in\Delta^l\times\mathcal{A}$ and every $\delta$, by the definition of the aggregate state drift and $0\le\beta\le B$ (definition of a transition-rate family), $|b^\delta(\Sigma,a)|\le\sum_{\sigma\neq\delta}\bigl(\Sigma^\sigma\beta(\sigma,\delta,\Sigma,a)+\Sigma^\delta\beta(\delta,\sigma,\Sigma,a)\bigr)\le B\sum_{\sigma\neq\delta}\Sigma^\sigma+(l-1)B\,\Sigma^\delta\le lB\le2(l-1)B$, since the coordinates of $\Sigma$ are nonnegative with sum $1$ and $l\ge2$.

Step 0 (interval splitting and the forward co-state equation). Let $\varphi:[0,T]\to\mathbb{R}$ be continuous and $t\in[0,T]$. We claim

where the first two integrals are the Lebesgue integrals of the restrictions of $\varphi$ (continuous by claim 1 of restriction stability, hence measurable and integrable by claim 3 of the toolkit) and the conventions $\int_{[0,0]}=0$, $\int_{[T,T]}=0$ are in force. For $t=0$ and $t=T$ the identity is trivial (zero extension, claim 2 of the toolkit, identifies the remaining integral with $\int_{[0,T]}\varphi$). For $0<t<T$: by claim 2 of the toolkit, $\int_{[0,t]}\varphi=\int_{[0,T]}\mathbf{1}_{[0,t]}\varphi$ and $\int_{[t,T]}\varphi=\int_{[0,T]}\mathbf{1}_{[t,T]}\varphi$. Pointwise on $[0,T]$, $\mathbf{1}_{[0,t]}+\mathbf{1}_{[t,T]}=1+\mathbf{1}_{\{t\}}$, so $\mathbf{1}_{[0,t]}\varphi+\mathbf{1}_{[t,T]}\varphi=\varphi+\mathbf{1}_{\{t\}}\varphi$; the function $\mathbf{1}_{\{t\}}\varphi$ vanishes off the set $\{t\}=[t,t]$, which is $\lambda_{[0,T]}$-null (a degenerate closed interval has Lebesgue measure zero by the Lebesgue measure theorem, and the toolkit's measure is the restriction of Lebesgue measure), so its integral is $0$ by claim 2 of the null-set integral lemma. The claim follows by linearity of the integral.

Consequently, for every $\gamma$ and every $t\in[0,T]$,

Indeed, by clause 2 of the co-state definition, $P^\gamma_t=-\partial_\gamma\bar{G}(S_T)+\int_t^T\varphi^\gamma_s\,ds$ with $\varphi^\gamma_s=\sum_\delta\partial_\gamma\bar{b}^\delta(S_s,A_s)P^\delta_s-\partial_\gamma\bar{L}(S_s,A_s)=-\partial_\gamma\mathcal{H}_s(S_s,A_s)$, a continuous function of $s$ (asserted there), the integral being the Riemann integral of the restriction; at $t=T$ this gives the second identity of $(\star)$. By claim 3 of the toolkit the Riemann integral equals the Lebesgue integral over $[t,T]$, and by the splitting just proved, $P^\gamma_t-P^\gamma_0=\int_{[t,T]}\varphi^\gamma-\int_{[0,T]}\varphi^\gamma=-\int_{[0,t]}\varphi^\gamma=\int_{[0,t]}\partial_\gamma\mathcal{H}_s(S_s,A_s)\,ds$.

(a). Define, at every $(t,\omega)\in[0,T]\times\Omega$,

By the joint measurability lemma, the maps $(t,\omega)\mapsto\mathbf{1}_{\Omega_0}\Sigma^\gamma_t$ and $(t,\omega)\mapsto\mathbf{1}_{\Omega_0}\alpha^j_t$ are measurable with respect to $\mathcal{B}_{[0,T]}\otimes\mathcal{F}$; the maps $(t,\omega)\mapsto\mathbf{1}_{\Omega\setminus\Omega_0}(\omega)S^\gamma_t$ and $(t,\omega)\mapsto\mathbf{1}_{\Omega\setminus\Omega_0}(\omega)A^j_t$ are measurable as products of an indicator of a measurable rectangle-factor and a continuous (hence measurable) deterministic map, by measurability of continuous functions of measurable maps; so $\tilde{\Sigma}^\gamma$ and $\tilde{\alpha}^j$ are product-measurable. At every point, $(\tilde{\Sigma}_t,\tilde{\alpha}_t)\in\Delta^l\times\mathcal{A}$: on $\Omega_0$ by the facts recorded above, off $\Omega_0$ because $(S_t,A_t)\in\Delta^l\times\mathcal{A}$. The map $(t,x,a)\mapsto\mathcal{H}_t(x,a)$ on $[0,T]\times(U\cap U_c)\times V$ is sequentially continuous (finite sums of products of the continuous $\bar{L}$, $\bar{b}^\delta$ — continuous by the extension definitions and part (i) of the regularity lemma — and the continuous $t\mapsto P^\delta_t$), so by the composition lemma the map $(t,\omega)\mapsto\mathcal{H}_t(\tilde{\Sigma}_t,\tilde{\alpha}_t)$ is product-measurable; likewise $(t,\omega)\mapsto L(\tilde{\Sigma}_t,\tilde{\alpha}_t)$ ($L$ being sequentially continuous on $\Delta^l\times\mathbb{R}^m$ by clause 1 of the cost-data definition), and the deterministic $t\mapsto\mathcal{H}_t(S_t,A_t)$ and $t\mapsto\partial_\gamma\mathcal{H}_t(S_t,A_t)$ are continuous, hence measurable. Since $\mathbf{1}_{\Omega_0}\mathcal{D}_t=\mathbf{1}_{\Omega_0}\bigl(\mathcal{H}_t(\tilde{\Sigma}_t,\tilde{\alpha}_t)-\mathcal{H}_t(S_t,A_t)-\sum_\gamma\partial_\gamma\mathcal{H}_t(S_t,A_t)(\tilde{\Sigma}^\gamma_t-S^\gamma_t)\bigr)$ and $\mathbf{1}_{\Omega_0}L(\Sigma_t,\alpha_t)=\mathbf{1}_{\Omega_0}L(\tilde{\Sigma}_t,\tilde{\alpha}_t)$ pointwise, both maps in the assertion are product-measurable (products and sums of product-measurable maps, by the composition lemma). Each $\Sigma^\gamma_T$ is a random variable (clause (iv) of the existence theorem) and $\bar{G}$ is continuous on $U_c\supseteq\Delta^l$, so $\mathcal{D}_G$ is a random variable by the composition lemma.

For the bounds, let $(t,\omega)$ be a point with $\Sigma_t\in\Delta^l$ and $\alpha_t\in\mathcal{A}$. Then $|\mathcal{H}_t(\Sigma_t,\alpha_t)|\le|\bar{L}(\Sigma_t,\alpha_t)|+\sum_\delta|P^\delta_t||\bar{b}^\delta(\Sigma_t,\alpha_t)|\le C_{LG}+C_P\cdot2(l-1)B$, using $\bar{L}=L$, $\bar{b}=b$ and the drift bound recorded above; the same bound holds for $|\mathcal{H}_t(S_t,A_t)|$. Both coordinates $\Sigma^\gamma_t$ and $S^\gamma_t$ lie in $[0,1]$, so $|y^\gamma_t|\le1$ and $\sum_\gamma|\partial_\gamma\mathcal{H}_t(S_t,A_t)\,y^\gamma_t|\le C_\partial$. Hence $|\mathcal{D}_t|\le2\bigl(C_{LG}+2(l-1)BC_P\bigr)+C_\partial=C_{\mathcal{D}}$. Likewise, at every $\omega$ (using $\Sigma_T\in\Delta^l$, $\bar{G}=G$ on $\Delta^l$, and $(\star)$): $|\mathcal{D}_G|\le2C_{LG}+\sum_\gamma|P^\gamma_T||y^\gamma_T|\le2C_{LG}+C_P=C_{\mathcal{D}G}$. Both membership statements hold at every point of $[0,T]\times\Omega$, respectively $\Omega$, by the facts recorded at the outset.

(b). Fix $\omega\in\Omega_0$. For a real-valued $\mathcal{B}_{[0,T]}\otimes\mathcal{F}$-measurable map $f$, every $\omega$-section is $\mathcal{B}_{[0,T]}$-measurable: the sections clause of the Tonelli-Fubini theorem is stated for $[0,\infty]$-valued maps, and applies to the nonnegative parts $f^+=\max(f,0)$ and $f^-=\max(-f,0)$ — measurable by the composition lemma — and $f=f^+-f^-$, so the $\omega$-section of $f$ is the difference of the measurable $\omega$-sections of $f^+$ and $f^-$; applying this to the product-measurable maps of (a) (whose $\omega$-sections at the fixed $\omega\in\Omega_0$ coincide with $t\mapsto L(\Sigma_t(\omega),\alpha_t(\omega))$, $t\mapsto\mathcal{D}_t(\omega)$) and to $\mathbf{1}_{\Omega_0}M^\gamma$ (product-measurable by part (b) of the restricted-moments lemma), all paths appearing below are measurable on $[0,T]$; they are bounded (by $C_{LG}$, $C_{\mathcal{D}}$, and $K_M=2+2(l-1)BT$ respectively), hence Lebesgue integrable over every $[0,t]$ by monotonicity against constants (linearity and monotonicity).

The pathwise state and deviation equations. By the displayed definition of $M$ in the statement and $\mathbf{1}_{\Omega_0}(\omega)=1$: $\Sigma^\gamma_t(\omega)=\Sigma^\gamma_0(\omega)+\int_{[0,t]}b^\gamma(\Sigma_s(\omega),\alpha_s(\omega))\,ds+M^\gamma_t(\omega)$ for every $t$ and $\gamma$; the integrand path is measurable (as the section of $\mathbf{1}_{\Omega_0}b^\gamma(\Sigma_\cdot,\alpha_\cdot)$, product-measurable by the composition argument of (a) applied to the sequentially continuous $\bar{b}^\gamma$) and bounded by $2(l-1)B$. By clause 2 of the trajectory-pair definition and claim 3 of the toolkit, $S^\gamma_t=S^\gamma_0+\int_{[0,t]}b^\gamma(S_s,A_s)\,ds$. Subtracting, with $g^\gamma_s=b^\gamma(\Sigma_s(\omega),\alpha_s(\omega))-b^\gamma(S_s,A_s)$ (measurable in $s$, $|g^\gamma_s|\le4(l-1)B$):

Integration by parts. Fix $\gamma$. Apply integration by parts for indefinite Lebesgue integrals with $f_s=\partial_\gamma\mathcal{H}_s(S_s,A_s)$ (continuous, bounded by $C_\partial$, integrable), $u_0=P^\gamma_0$ — so that $u_t=P^\gamma_t$ by $(\star)$ — and with $g_s=g^\gamma_s$, $v_0=y^\gamma_0(\omega)$ — so that $v_t=y^\gamma_t(\omega)-M^\gamma_t(\omega)$ by $(\dagger)$. Part (ii) of that lemma gives

all integrals existing. Each of the four summands under the integral is bounded and measurable, hence integrable, so by linearity

Hamiltonian split. For every $s$, both $(\Sigma_s(\omega),\alpha_s(\omega))$ and $(S_s,A_s)$ lie in $\Delta^l\times\mathcal{A}$, so by the definition of $\mathcal{H}$ and the extension agreements,

and pointwise $\mathcal{H}_s(\Sigma_s,\alpha_s)-\mathcal{H}_s(S_s,A_s)=\mathcal{D}_s+\sum_\gamma\partial_\gamma\mathcal{H}_s(S_s,A_s)y^\gamma_s$. Also, using $\bar{G}=G$ on $\Delta^l$ and $(\star)$,

Integrating the first display over $[0,T]$ (each summand integrable), substituting $(\ddagger)$ summed over $\gamma$, and adding the terminal display:

the terms $\sum_\gamma P^\gamma_Ty^\gamma_T$ and $\pm\int\sum_\gamma\partial_\gamma\mathcal{H}\,y^\gamma$ cancelling. By the definition of the mean-field cost and claim 3 of the toolkit, $\int_{[0,T]}L(S_t,A_t)\,dt+G(S_T)=J^{MF}$, so the display of (b) follows by rearrangement.

Finiteness of $J^N[h]$. Define $w=\mathbf{1}_{\Omega_0}\cdot\bigl(\int_{[0,T]}L(\Sigma_t,\alpha_t)\,dt+G(\Sigma_T)\bigr)$, a well-defined function on $\Omega$ (the integral existing at every $\omega\in\Omega_0$ as above, the value being $0$ off $\Omega_0$); $w$ equals $\omega\mapsto\int_{[0,T]}\mathbf{1}_{\Omega_0}L(\Sigma_t,\alpha_t)\,dt+\mathbf{1}_{\Omega_0}G(\Sigma_T)$, which is a random variable: the integrand is product-measurable and bounded by (a), so writing it as a difference of its nonnegative and negative parts and applying the Tonelli clause of the Tonelli-Fubini theorem to each part shows that $\omega\mapsto\int_{[0,T]}\mathbf{1}_{\Omega_0}L(\Sigma_t,\alpha_t)\,dt$ is a difference of measurable maps, hence measurable by the composition lemma, and $\mathbf{1}_{\Omega_0}G(\Sigma_T)$ is a random variable as in (a); moreover $|w|\le C_{LG}(T+1)$. By the definition of the $N$-agent cost, $J^N[h]$ is the expectation of the cost variable of clause (v) of the existence theorem, an extended-real-valued variable bounded below whose expectation is well defined in $(-\infty,+\infty]$; call it $w'$. At every point of the probability-one event $\Omega_0$, $w'$ coincides with $w$ (both are the same pathwise integral plus $G(\Sigma_T)$, the section being measurable at every $\omega\in\Omega_0$ as shown above). For nonnegative extended-real-valued variables $U,V$ agreeing off an event $E'$ of probability zero one has $\mathbb{E}[U]=\mathbb{E}[V]$: by monotone convergence $\mathbb{E}[U\mathbf{1}_{E'}]$ is the limit of $\mathbb{E}[\min(U,n)\mathbf{1}_{E'}]\le n\,\mathbb{E}[\mathbf{1}_{E'}]=0$, so $\mathbb{E}[U]=\mathbb{E}[U\mathbf{1}_{\Omega\setminus E'}]=\mathbb{E}[V\mathbf{1}_{\Omega\setminus E'}]=\mathbb{E}[V]$. Applying this to the nonnegative and negative parts of $w'$ and $w$ (which agree off $\Omega\setminus\Omega_0$) gives $J^N[h]=\mathbb{E}[w']=\mathbb{E}[w]$, a finite real number since $|w|\le C_{LG}(T+1)$.

(c). By (a) the map $(t,\omega)\mapsto\mathbf{1}_{\Omega_0}\mathcal{D}_t$ is product-measurable and bounded by $C_{\mathcal{D}}$; on the finite product of $([0,T],\mathcal{B}_{[0,T]},\lambda_{[0,T]})$ and the probability space, the Tonelli clause of the Tonelli-Fubini theorem applied to the absolute value shows the map is integrable for the product measure (its integral being at most $C_{\mathcal{D}}\cdot T$), so the Fubini clause applies: $t\mapsto\mathbb{E}[\mathbf{1}_{\Omega_0}\mathcal{D}_t]$ is measurable and (being bounded by $C_{\mathcal{D}}$) bounded, and

Next, $\mathbb{E}[\mathbf{1}_{\Omega_0}M^\gamma_t]=0$ for every $t$ and $\gamma$: the constant function $T$ is a stopping time of $(\mathcal{F}^{\mathrm{sys}}_t)_{t\in[0,T]}$ (claim 1 of the stopping-time toolkit), so claim 4 of the covariation lemma with $\tau\equiv T$, $r=0$, $D=\Omega$ gives $\mathbb{E}[\mathbf{1}_{\Omega_0}M^\gamma_{\min(t,T)}]=\mathbb{E}[\mathbf{1}_{\Omega_0}M^\gamma_{\min(0,T)}]$, and $M^\gamma_0=0$ identically (the defining display at $t=0$, with the convention $\int_{[0,0]}=0$). Applying the same Tonelli-then-Fubini step to the bounded product-measurable map $(t,\omega)\mapsto\partial_\gamma\mathcal{H}_t(S_t,A_t)\,\mathbf{1}_{\Omega_0}M^\gamma_t$ (bounded by $C_\partial K_M$ with $K_M=2+2(l-1)BT$ of part (b) of the restricted-moments lemma, which also provides the product-measurability of $\mathbf{1}_{\Omega_0}M^\gamma$; note $\mathbf{1}_{\Omega_0}(\omega)\int_{[0,T]}\sum_\gamma\partial_\gamma\mathcal{H}_t(S_t,A_t)M^\gamma_t(\omega)\,dt=\int_{[0,T]}\sum_\gamma\partial_\gamma\mathcal{H}_t(S_t,A_t)\,\mathbf{1}_{\Omega_0}(\omega)M^\gamma_t(\omega)\,dt$ at every $\omega\in\Omega_0$, and both sides vanish off $\Omega_0$, with $R^M$ extended by $0$ there) then gives $\mathbb{E}\bigl[\mathbf{1}_{\Omega_0}\int_{[0,T]}\sum_\gamma\partial_\gamma\mathcal{H}_t(S_t,A_t)M^\gamma_t\,dt\bigr]=\int_{[0,T]}\sum_\gamma\partial_\gamma\mathcal{H}_t(S_t,A_t)\,\mathbb{E}[\mathbf{1}_{\Omega_0}M^\gamma_t]\,dt=0$, and $\mathbb{E}[\mathbf{1}_{\Omega_0}R^M]=-\sum_\gamma P^\gamma_T\,\mathbb{E}[\mathbf{1}_{\Omega_0}M^\gamma_T]+0=0$.

Multiply the identity of (b) by $\mathbf{1}_{\Omega_0}$, take expectations (every term is a bounded random variable by (a) and the above), and use: $\mathbb{E}[w]=J^N[h]$; $\mathbb{E}[\mathbf{1}_{\Omega_0}]\,J^{MF}=J^{MF}$ and, for any bounded random variable $X$, $\mathbb{E}[\mathbf{1}_{\Omega_0}X]=\mathbb{E}[X]$ (the difference $\mathbf{1}_{\Omega\setminus\Omega_0}X$ vanishes off a null event); $\mathbb{E}[y^\gamma_0]=\mathbb{E}[\Sigma^\gamma_0]-S^\gamma_0=\zeta^\gamma_N/N$. This yields

and multiplying by $N$ gives (c), the factor $N$ passing inside the integral and the expectations by linearity.

(d). Fix $\omega\in\Omega$ and $t\in[0,T]$; then $(\Sigma_t(\omega),\alpha_t(\omega))\in\Delta^l\times\mathcal{A}$ by the facts recorded at the outset. By part (b) of the first-order lemma (its hypotheses — $\mathcal{A}$ compact and, for part (b), convex — hold),

By conclusion (d) of the growth lemma under (A), (H1), (U) — the function there being $a\mapsto\mathcal{H}_t(S_t,a)$, as identified in the first-order lemma — $\mathcal{H}_t(S_t,\alpha_t)-\mathcal{H}_t(S_t,A_t)\ge r_0|\alpha_t-A_t|^2$. From $\bigl(\sqrt{r_0}\,|\alpha_t-A_t|-(C_1/\sqrt{r_0})|y_t|\bigr)^2\ge0$ one gets $C_1|y_t||\alpha_t-A_t|\le\tfrac{r_0}{2}|\alpha_t-A_t|^2+\tfrac{C_1^2}{2r_0}|y_t|^2$. Combining,

and multiplying by $N$, with $N|\alpha_t-A_t|^2=|\mathfrak{a}_t|^2$ and $N|y_t|^2=|\mathfrak{s}_t|^2$, gives the running bound. For the terminal bound, at every $\omega$: $\Sigma_T\in\Delta^l$, so the second display of part (b) of the first-order lemma gives $|\mathcal{D}_G|\le\tfrac{l\,K_c}{2}|y_T|^2$, whence $N\mathcal{D}_G\ge-\tfrac{lK_c}{2}|\mathfrak{s}_T|^2$.

(e). By the joint measurability lemma and the composition lemma, $(t,\omega)\mapsto\mathbf{1}_{\Omega_0}|\mathfrak{a}_t|^2$ and $(t,\omega)\mapsto\mathbf{1}_{\Omega_0}|\mathfrak{s}_t|^2$ are product-measurable; they are bounded (for the fixed $N$) by $4R_{\mathcal{A}}^2N$ and $4N$ respectively, where $R_{\mathcal{A}}$ is a bound for the norms of points of the compact, hence bounded, set $\mathcal{A}$ (Heine-Borel) and $|\mathfrak{s}_t|\le2\sqrt{N}$ as in the coordinate argument of (a). By Tonelli-Fubini, $t\mapsto\mathbb{E}[\mathbf{1}_{\Omega_0}|\mathfrak{a}_t|^2]$ and $t\mapsto\mathbb{E}[\mathbf{1}_{\Omega_0}|\mathfrak{s}_t|^2]$ are measurable and bounded, so all integrals in (e) exist and are finite. By (d) and monotonicity of the expectation, for every $t$,

and $\mathbb{E}[\mathbf{1}_{\Omega_0}N\mathcal{D}_G]\ge-\tfrac{lK_c}{2}\mathbb{E}[\mathbf{1}_{\Omega_0}|\mathfrak{s}_T|^2]$. Integrating the first inequality over $[0,T]$ (monotonicity and linearity of the interval integral) and inserting both into the identity of (c) yields

which is (e) upon rearrangement. $\blacksquare$