Proof of Second-Order Expansion of the N-Agent Cost about a Stationary Mean-Field Trajectory

Throughout, write $\mathbf{1}=\mathbf{1}_{\Omega_0}$, $w_t=(\Sigma_t,\alpha_t)-(S_t,A_t)\in\mathbb{R}^{l+m}$ with components $w^i_t$, so that $w_t=N^{-1/2}(\mathfrak{s}_t,\mathfrak{a}_t)$ and $\rho_t=|w_t|$, and abbreviate $m_t=|\mathfrak{s}_t|^2+|\mathfrak{a}_t|^2=N\rho_t^2$. Let $b$ denote the aggregate state drift of $\beta$, which agrees with the extended aggregate state drift $\bar{b}$ on $\Delta^l\times\mathcal{A}$ by part (i) of the drift regularity lemma; every state-control point at which this agreement is invoked below lies in $\Delta^l\times\mathcal{A}$, both $\alpha_t$ and $A_t$ taking values in $\mathcal{A}$.

Step 0 (measurability conventions). Every $\Sigma_t$ lies in $\Delta^l$ (each agent occupies exactly one state, by the derived notation of the solution definition) and every $\alpha_t$ lies in $\mathcal{A}$ (the policy being $\mathcal{A}$-valued), at every point of $\Omega$. Define the modified state-control map $Z$ on $[0,T]\times\Omega$ by $Z=(\Sigma_t,\alpha_t)$ for $\omega\in\Omega_0$ and $Z=(e_1,a_0)$ off $\Omega_0$, where $e_1=(1,0,\dots,0)\in\Delta^l$ and $a_0$ is a fixed point of the nonempty control set $\mathcal{A}$, so that $Z$ takes values in $\Delta^l\times\mathcal{A}$; by the joint measurability lemma each component of $Z$ is product-measurable with respect to the product $\sigma$-algebra of the trace Borel $\sigma$-algebra and $\mathcal{F}$. The map $(t,\omega)\mapsto(S_t,A_t)$ is product-measurable, being the composition of $(t,\omega)\mapsto t$ with the continuous trajectory components, by measurability of sequentially continuous functions of measurable maps. Every integrand appearing below is built from these maps, the co-state, and restrictions of $\bar{L}$ to $\Delta^l\times\mathbb{R}^m$, of $\bar{b}^\delta$ to $\Delta^l\times V$ (which contains the range $\Delta^l\times\mathcal{A}$ of $Z$, $\mathcal{A}$ being a subset of $V$ by the extension definition), and of $\bar{G}$ to $\Delta^l$, together with their first and second partial derivatives - all sequentially continuous on their domains, continuity of the $C^1$ maps and of their partials being part of the extension definitions and of the drift regularity lemma - together with sums, products, and the moduli. Compositions with sequentially continuous functions preserve product-measurability by the composition lemma; sums and products are compositions with the continuous arithmetic operations; and each modulus is nondecreasing on $[0,\infty)$, so its sublevel sets are intervals, hence Borel, and $\omega_L(\rho_t)$, $\omega_b(\rho_t)$, $\omega_G$-compositions are product-measurable. Consequently every integrand below is product-measurable after multiplication by $\mathbf{1}$ (equivalently, after substituting $Z$); since $\Omega_0$ has probability $1$, no expectation is affected, and we use this silently. Sections and partial integrals of nonnegative product-measurable maps are measurable by the Tonelli theorem, whose applications here are on the product of two finite, hence $\sigma$-finite, measure spaces (the restricted Lebesgue measure of $[0,T]$ has total mass $T$ by the toolkit, and $(\Omega,\mathcal{F},P)$ is a probability space).

Step 1 (part (a)). Each modulus is nondecreasing because the supremum is over a set that grows with $u$, and nonnegative. By clause 3 of the cost extension, any two values of a second partial of $\bar{L}$ (or of $\bar{G}$) differ by at most $2K_c$, so $\omega_L\le2K_c$ and $\omega_G\le2K_c$; by part (iii) of the drift regularity lemma, $|\partial_j\partial_i\bar{b}^\gamma|\le3lK$ on $\Delta^l\times V$, so $\omega_b\le6lK$. Given $\varepsilon>0$: clause 4 of the cost extension yields $\delta>0$ making all oscillations of the second partials of $\bar{L}$ and $\bar{G}$ at distance at most $\delta$ no larger than $\varepsilon$, whence $\omega_L(u)\le\varepsilon$ and $\omega_G(u)\le\varepsilon$ for $u\in[0,\delta]$; part (iii) of the drift regularity lemma yields the same for $\omega_b$.

Step 2 (pointwise Taylor expansions). Fix $(t,\omega)$. The segment from $(S_t,A_t)$ to $(\Sigma_t,\alpha_t)$ lies in $\Delta^l\times V$: a convex combination of two simplex points has nonnegative entries summing to $1$, and both $A_t$ and $\alpha_t$ lie in $\mathcal{A}\subseteq V$, the set $V$ being convex by the extension definition. Hence the segment lies in $\Delta^l\times\mathbb{R}^m$ - the domain of the $\omega_L$ supremum - and in the open sets $W\times\mathbb{R}^m$ and $U\times V$; likewise the segment from $S_T$ to $\Sigma_T$ lies in $\Delta^l\subset W$. By the definition of the moduli, along these segments the oscillation of each second partial of $\bar{L}$, $\bar{b}^\delta$, $\bar{G}$ against its value at the base point is at most $\omega_L(\rho_t)$, $\omega_b(\rho_t)$, $\omega_G(d(\Sigma_T,S_T))$ respectively. Part (iii) of the Taylor expansion lemma (with $n=l+m$ for $\bar{L}$ and $\bar{b}^\delta$, whose iterated-partial regularity is supplied by clause 2 of the extensions and by part (i) of the drift regularity lemma, and $n=l$ for $\bar{G}$), together with clause 1 of both extensions and the restriction clause of the drift regularity lemma to rewrite values on the simplex product in terms of $L$, $b$, $G$, gives

where all sums over $i,j$ run over $\{1,\dots,l+m\}$ and those over $\gamma,\delta$ over $\{1,\dots,l\}$.

Step 3 (integrability and part (b)). Each map $t\mapsto\partial_i\bar{L}(S_t,A_t)$ is continuous relative to $[0,T]$ with the metric of the real line (a composition of continuous maps, the Euclidean and metric notions of continuity agreeing for real-valued maps by claim 1 of the continuity agreement lemma) and therefore attains a maximum and a minimum on $[0,T]$ by the extreme value theorem; let $C_1$ bound them all in absolute value. By the a priori second-moment bound and the hypothesis $\mathcal{A}_2<\infty$, there is $C_2<\infty$ with $\mathbb{E}[|\mathfrak{s}_t|^2]\le C_2$ for all $t$, so $\int_{[0,T]}\mathbb{E}[m_t]\,dt\le TC_2+\mathcal{A}_2<\infty$; also $\mathbb{E}[|\mathfrak{s}_T|^2]\le4N$ by part (a) of that lemma. Since $|w^i_t|\le\rho_t\le\tfrac{1}{2}(1+\rho_t^2)$ and $N\rho_t^2=m_t$, we get $\int_{[0,T]}\mathbb{E}[\rho_t]\,dt<\infty$. Each of the three pieces on the right of (1) then has finite $\int_{[0,T]}\mathbb{E}|\cdot|\,dt$: the linear piece is at most $C_1\sum_i|w^i_t|\le C_1(l+m)\rho_t$; the quadratic piece at most $\tfrac{1}{2}(l+m)^2K_c\rho_t^2$ (clause 3); and $|r^L_t|\le(l+m)K_c\rho_t^2$ since $\omega_L\le2K_c$. As $t\mapsto L(S_t,A_t)$ is continuous and bounded, $\int_{[0,T]}\mathbb{E}|L(\Sigma_t,\alpha_t)|\,dt<\infty$. Similarly, using (3) with the pointwise bounds $|\partial_\gamma\bar{G}(S_T)|$ finite, $\omega_G\le2K_c$, and $\mathbb{E}|\mathfrak{s}_T|^2\le4N$, the terminal difference is integrable. Hence the random variable $\int_{[0,T]}L(\Sigma_t,\alpha_t)\,dt+G(\Sigma_T)$, whose expectation defines $J^N[h]$ in $(-\infty,+\infty]$ by the $N$-agent cost definition, is integrable, $J^N[h]$ is finite, and by linearity of the expectation and the Fubini theorem,

where the deterministic parts of $J^{MF}$ (mean-field cost, a Riemann integral) were converted to Lebesgue integrals by claim 3 of the integral toolkit on a compact interval, the integrand $t\mapsto L(S_t,A_t)$ being continuous as recorded in the mean-field cost definition. Splitting (4) by (1) and (3) into linear, quadratic, and remainder groups is legitimate since each group is absolutely integrable. Eligibility of $((\mathfrak{s}),(\mathfrak{a}))$ for the fluctuation linear-quadratic cost: requirement (i) holds with $\Omega_1=\Omega_0$ (Step 0), (ii) is $\int_{[0,T]}\mathbb{E}[m_t]dt<\infty$, and (iii) is $\mathbb{E}[|\mathfrak{s}_T|^2]\le4N$. This proves (b), the finiteness of the expressions in (c) being contained in the estimates above and below.

Step 4 (linear terms). Set $\psi^\gamma(t)=\mathbb{E}[\Sigma^\gamma_t]-S^\gamma_t=\mathbb{E}[w^\gamma_t]$. By part (b) of the martingale decomposition theorem, $M^\gamma_t=\Sigma^\gamma_t-\Sigma^\gamma_0-\int_{[0,t]}\mathbf{1}\,b^\gamma(\Sigma_s,\alpha_s)ds$ is a square-integrable martingale with $M^\gamma_0=0$, so its defining property with $r=0$ and $D=\Omega$ gives $\mathbb{E}[M^\gamma_t]=\mathbb{E}[M^\gamma_0]=0$. Taking expectations and applying the Fubini theorem to the bounded integrand $\mathbf{1}\,b^\gamma(\Sigma_s,\alpha_s)$ (bounded by $2(l-1)B$ by part (a) of the decomposition theorem; dropping the indicator changes no expectation, $\Omega_0$ having probability $1$), and using condition 2 of the mean-field trajectory pair (its Riemann integral converted to a Lebesgue integral over $[0,t]$ by claim 3 of the integral toolkit, applied for $t>0$ to the restriction of the continuous integrand, continuous by claim 1 of restriction stability, the case $t=0$ being trivial),

with $\hat{g}^\gamma$ measurable (Tonelli) and bounded. By clause 2 of the stationary co-state definition and additivity of the Riemann integral, with the continuous function $p^\gamma(s)=\partial_\gamma\bar{L}(S_s,A_s)-\sum_{\delta}\partial_\gamma\bar{b}^\delta(S_s,A_s)P^\delta_s$ we have $P^\gamma_t=P^\gamma_0+\int_{[0,t]}p^\gamma(s)ds$ for all $t$ (additivity applied for $0<t<T$, the cases $t=0$ and $t=T$ holding by the zero-integral conventions of clause 2 of the co-state definition; the Riemann integrals over $[0,t]$ are converted to Lebesgue integrals as in the preceding conversion). The integration by parts lemma applied to $u=P^\gamma$, $v=\psi^\gamma$ gives

The linear group of (4) is

where the state part was written with the expectation inside the time integral by the Fubini theorem. By clause 2 of the co-state definition at $t=T$, $\partial_\gamma\bar{G}(S_T)=-P^\gamma_T$, so the terminal term equals $-\sum_\gamma P^\gamma_T\psi^\gamma(T)$. By the stationarity clause 3, pointwise in $(t,\omega)$, $\sum_j\partial_{l+j}\bar{L}(S_t,A_t)w^{l+j}_t=\sum_j\sum_\delta\partial_{l+j}\bar{b}^\delta(S_t,A_t)P^\delta_t\,w^{l+j}_t$. Substituting these and (5), and observing that $\partial_\gamma\bar{L}(S_t,A_t)=p^\gamma(t)+\sum_\delta\partial_\gamma\bar{b}^\delta(S_t,A_t)P^\delta_t$ makes the $\int\sum_\gamma p^\gamma\psi^\gamma$ terms cancel,

Moving the expectation back inside the first integral (Fubini), combining the three integrals under one $\mathbb{E}\int$ (each absolutely integrable by Step 3 and boundedness of the coefficients), and using $\sum_\gamma\partial_\gamma\bar{b}^\delta w^\gamma+\sum_j\partial_{l+j}\bar{b}^\delta w^{l+j}=\sum_{i=1}^{l+m}\partial_i\bar{b}^\delta w^i$ together with the definition of $\hat{g}^\delta$,

and by the Taylor identity (2),

Step 5 (assembly and remainder bound). Substituting (6) and the quadratic and remainder groups of (1) and (3) into (4),

Multiply by $N$ and use $w_t=N^{-1/2}(\mathfrak{s}_t,\mathfrak{a}_t)$: the first term becomes $-\sum_\gamma P^\gamma_0\zeta^\gamma_N$ since $\zeta^\gamma_N=N\psi^\gamma(0)$; the two quadratic terms become exactly $LQG[(\mathfrak{s}),(\mathfrak{a})]$ with the fluctuation Hessian coefficients $H_{ij}(t)$ and $F_{\gamma\delta}$ of the fluctuation linear-quadratic cost evaluated at $z_t=(\mathfrak{s}_t,\mathfrak{a}_t)$; and thus the identity of part (c) holds with

By the bounds in (1)-(3) and $\sum_\delta|P^\delta_t|\le C_P$, pointwise

and $N|r^G|\le\tfrac{1}{2}\,l\,\omega_G(d(\Sigma_T,S_T))\,|\mathfrak{s}_T|^2$ since $N|\Sigma_T-S_T|^2=|\mathfrak{s}_T|^2$. Taking absolute values inside the expectations (triangle inequality) and applying the Tonelli theorem to the nonnegative product-measurable dominating integrand gives

which is finite because the moduli are bounded (part (a)) and $\int_{[0,T]}\mathbb{E}[m_t]dt<\infty$, $\mathbb{E}[|\mathfrak{s}_T|^2]<\infty$ (Step 3). This completes the proof of (c).