Proof of Second-Order Expansion of the N-Agent Cost about a Stationary Mean-Field Trajectory

Throughout, write $\mathbf{1}=\mathbf{1}_{\Omega_0}$, $w_t=(\Sigma_t,\alpha_t)-(S_t,A_t)\in\mathbb{R}^{l+m}$ with components $w^i_t$, so that $w_t=N^{-1/2}(\mathfrak{s}_t,\mathfrak{a}_t)$ and $\rho_t=|w_t|$, and abbreviate $m_t=|\mathfrak{s}_t|^2+|\mathfrak{a}_t|^2=N\rho_t^2$. Let $b$ denote the aggregate state drift of $\beta$, which agrees with the extended aggregate state drift $\bar{b}$ on $\Delta^l\times\mathbb{R}^m$ by part (i) of the drift regularity lemma.

Step 0 (measurability conventions). Every $\Sigma_t$ lies in $\Delta^l$ and every $\alpha_t$ in $\mathbb{R}^m$, at every point of $\Omega$ (each agent occupies exactly one state, by the derived notation of the solution definition). Define the modified state-control map $Z$ on $[0,T]\times\Omega$ by $Z=(\Sigma_t,\alpha_t)$ for $\omega\in\Omega_0$ and $Z=(e_1,0)$ off $\Omega_0$, where $e_1=(1,0,\dots,0)\in\Delta^l$; by the joint measurability lemma each component of $Z$ is product-measurable with respect to the product $\sigma$-algebra of the trace Borel $\sigma$-algebra and $\mathcal{F}$. The map $(t,\omega)\mapsto(S_t,A_t)$ is product-measurable, being the composition of $(t,\omega)\mapsto t$ with the continuous trajectory components, by measurability of sequentially continuous functions of measurable maps. Every integrand appearing below is built from these maps, the co-state, and restrictions to $\Delta^l\times\mathbb{R}^m$ (or $\Delta^l$) of $\bar{L}$, $\bar{b}^\delta$, $\bar{G}$ and their first and second partial derivatives - all sequentially continuous on their domains, continuity of the $C^1$ maps and of their partials being part of the extension definitions and of the drift regularity lemma - together with sums, products, and the moduli. Compositions with sequentially continuous functions preserve product-measurability by the composition lemma; sums and products are compositions with the continuous arithmetic operations; and each modulus is nondecreasing on $[0,\infty)$, so its sublevel sets are intervals, hence Borel, and $\omega_L(\rho_t)$, $\omega_b(\rho_t)$, $\omega_G$-compositions are product-measurable. Consequently every integrand below is product-measurable after multiplication by $\mathbf{1}$ (equivalently, after substituting $Z$); since $\Omega_0$ has probability $1$, no expectation is affected, and we use this silently. Sections and partial integrals of nonnegative product-measurable maps are measurable by the Tonelli theorem, whose applications here are on the product of two finite, hence $\sigma$-finite, measure spaces (the restricted Lebesgue measure of $[0,T]$ has total mass $T$ by the toolkit, and $(\Omega,\mathcal{F},P)$ is a probability space).

Step 1 (part (a)). Each modulus is nondecreasing because the supremum is over a set that grows with $u$, and nonnegative. By clause 3 of the cost extension, any two values of a second partial of $\bar{L}$ (or of $\bar{G}$) differ by at most $2K_c$, so $\omega_L\le2K_c$ and $\omega_G\le2K_c$; by part (iii) of the drift regularity lemma, $|\partial_j\partial_i\bar{b}^\gamma|\le3lK$ on $\Delta^l\times\mathbb{R}^m$, so $\omega_b\le6lK$. Given $\varepsilon>0$: clause 4 of the cost extension yields $\delta>0$ making all oscillations of the second partials of $\bar{L}$ and $\bar{G}$ at distance at most $\delta$ no larger than $\varepsilon$, whence $\omega_L(u)\le\varepsilon$ and $\omega_G(u)\le\varepsilon$ for $u\in[0,\delta]$; part (iii) of the drift regularity lemma yields the same for $\omega_b$.

Step 2 (pointwise Taylor expansions). Fix $(t,\omega)$. The segment from $(S_t,A_t)$ to $(\Sigma_t,\alpha_t)$ lies in $\Delta^l\times\mathbb{R}^m$ (a convex combination of two simplex points has nonnegative entries summing to $1$), hence in the open sets $V\times\mathbb{R}^m$ and $U\times\mathbb{R}^m$; likewise the segment from $S_T$ to $\Sigma_T$ lies in $\Delta^l\subset V$. By the definition of the moduli, along these segments the oscillation of each second partial of $\bar{L}$, $\bar{b}^\delta$, $\bar{G}$ against its value at the base point is at most $\omega_L(\rho_t)$, $\omega_b(\rho_t)$, $\omega_G(d(\Sigma_T,S_T))$ respectively. Part (iii) of the Taylor expansion lemma (with $n=l+m$ for $\bar{L}$ and $\bar{b}^\delta$, whose iterated-partial regularity is supplied by clause 2 of the extensions and by part (i) of the drift regularity lemma, and $n=l$ for $\bar{G}$), together with clause 1 of both extensions and the restriction clause of the drift regularity lemma to rewrite values on the simplex product in terms of $L$, $b$, $G$, gives

where all sums over $i,j$ run over $\{1,\dots,l+m\}$ and those over $\gamma,\delta$ over $\{1,\dots,l\}$.

Step 3 (integrability and part (b)). Each map $t\mapsto\partial_i\bar{L}(S_t,A_t)$ is continuous (a composition of continuous maps) and hence bounded; let $C_1$ bound them all. By the a priori second-moment bound and the hypothesis $\mathcal{A}<\infty$, there is $C_2<\infty$ with $\mathbb{E}[|\mathfrak{s}_t|^2]\le C_2$ for all $t$, so $\int_{[0,T]}\mathbb{E}[m_t]\,dt\le TC_2+\mathcal{A}<\infty$; also $\mathbb{E}[|\mathfrak{s}_T|^2]\le4N$ by part (a) of that lemma. Since $|w^i_t|\le\rho_t\le\tfrac{1}{2}(1+\rho_t^2)$ and $N\rho_t^2=m_t$, we get $\int_{[0,T]}\mathbb{E}[\rho_t]\,dt<\infty$. Each of the three pieces on the right of (1) then has finite $\int_{[0,T]}\mathbb{E}|\cdot|\,dt$: the linear piece is at most $C_1\sum_i|w^i_t|\le C_1(l+m)\rho_t$; the quadratic piece at most $\tfrac{1}{2}(l+m)^2K_c\rho_t^2$ (clause 3); and $|r^L_t|\le(l+m)K_c\rho_t^2$ since $\omega_L\le2K_c$. As $t\mapsto L(S_t,A_t)$ is continuous and bounded, $\int_{[0,T]}\mathbb{E}|L(\Sigma_t,\alpha_t)|\,dt<\infty$. Similarly, using (3) with the pointwise bounds $|\partial_\gamma\bar{G}(S_T)|$ finite, $\omega_G\le2K_c$, and $\mathbb{E}|\mathfrak{s}_T|^2\le4N$, the terminal difference is integrable. Hence the random variable $\int_{[0,T]}L(\Sigma_t,\alpha_t)\,dt+G(\Sigma_T)$, whose expectation defines $J^N[h]$ in $(-\infty,+\infty]$ by the $N$-agent cost definition, is integrable, $J^N[h]$ is finite, and by linearity of the expectation and the Fubini theorem,

where the deterministic parts of $J^{MF}$ (mean-field cost, a Riemann integral) were converted by the agreement of the Riemann and Lebesgue integrals. Splitting (4) by (1) and (3) into linear, quadratic, and remainder groups is legitimate since each group is absolutely integrable. Eligibility of $((\mathfrak{s}),(\mathfrak{a}))$ for the fluctuation linear-quadratic cost: requirement (i) holds with $\Omega_1=\Omega_0$ (Step 0), (ii) is $\int_{[0,T]}\mathbb{E}[m_t]dt<\infty$, and (iii) is $\mathbb{E}[|\mathfrak{s}_T|^2]\le4N$. This proves (b), the finiteness of the expressions in (c) being contained in the estimates above and below.

Step 4 (linear terms). Set $\psi^\gamma(t)=\mathbb{E}[\Sigma^\gamma_t]-S^\gamma_t=\mathbb{E}[w^\gamma_t]$. By part (b) of the martingale decomposition theorem, $M^\gamma_t=\Sigma^\gamma_t-\Sigma^\gamma_0-\int_{[0,t]}b^\gamma(\Sigma_s,\alpha_s)ds$ is a square-integrable martingale with $M^\gamma_0=0$, so its defining property with $r=0$ and $D=\Omega$ gives $\mathbb{E}[M^\gamma_t]=\mathbb{E}[M^\gamma_0]=0$. Taking expectations and applying the Fubini theorem to the bounded integrand $b^\gamma(\Sigma_s,\alpha_s)$ (bounded by $2(l-1)B$ by part (a) of the decomposition theorem), and using condition 2 of the mean-field trajectory pair,

with $\hat{g}^\gamma$ measurable (Tonelli) and bounded. By clause 2 of the stationary co-state definition and additivity of the Riemann integral, with the continuous function $p^\gamma(s)=\partial_\gamma\bar{L}(S_s,A_s)-\sum_{\delta}\partial_\gamma\bar{b}^\delta(S_s,A_s)P^\delta_s$ we have $P^\gamma_t=P^\gamma_0+\int_{[0,t]}p^\gamma(s)ds$ for all $t$ (Riemann and Lebesgue integrals agreeing for continuous integrands). The integration by parts lemma applied to $u=P^\gamma$, $v=\psi^\gamma$ gives

The linear group of (4) is

where the state part was written with the expectation inside the time integral by the Fubini theorem. By clause 2 of the co-state definition at $t=T$, $\partial_\gamma\bar{G}(S_T)=-P^\gamma_T$, so the terminal term equals $-\sum_\gamma P^\gamma_T\psi^\gamma(T)$. By the stationarity clause 3, pointwise in $(t,\omega)$, $\sum_j\partial_{l+j}\bar{L}(S_t,A_t)w^{l+j}_t=\sum_j\sum_\delta\partial_{l+j}\bar{b}^\delta(S_t,A_t)P^\delta_t\,w^{l+j}_t$. Substituting these and (5), and observing that $\partial_\gamma\bar{L}(S_t,A_t)=p^\gamma(t)+\sum_\delta\partial_\gamma\bar{b}^\delta(S_t,A_t)P^\delta_t$ makes the $\int\sum_\gamma p^\gamma\psi^\gamma$ terms cancel,

Moving the expectation back inside the first integral (Fubini), combining the three integrals under one $\mathbb{E}\int$ (each absolutely integrable by Step 3 and boundedness of the coefficients), and using $\sum_\gamma\partial_\gamma\bar{b}^\delta w^\gamma+\sum_j\partial_{l+j}\bar{b}^\delta w^{l+j}=\sum_{i=1}^{l+m}\partial_i\bar{b}^\delta w^i$ together with the definition of $\hat{g}^\delta$,

and by the Taylor identity (2),

Step 5 (assembly and remainder bound). Substituting (6) and the quadratic and remainder groups of (1) and (3) into (4),

Multiply by $N$ and use $w_t=N^{-1/2}(\mathfrak{s}_t,\mathfrak{a}_t)$: the first term becomes $-\sum_\gamma P^\gamma_0\zeta^\gamma_N$ since $\zeta^\gamma_N=N\psi^\gamma(0)$; the two quadratic terms become exactly $LQG[(\mathfrak{s}),(\mathfrak{a})]$ with the fluctuation Hessian coefficients $H_{ij}(t)$ and $F_{\gamma\delta}$ of the fluctuation linear-quadratic cost evaluated at $z_t=(\mathfrak{s}_t,\mathfrak{a}_t)$; and thus the identity of part (c) holds with

By the bounds in (1)-(3) and $\sum_\delta|P^\delta_t|\le C_P$, pointwise

and $N|r^G|\le\tfrac{1}{2}\,l\,\omega_G(d(\Sigma_T,S_T))\,|\mathfrak{s}_T|^2$ since $N|\Sigma_T-S_T|^2=|\mathfrak{s}_T|^2$. Taking absolute values inside the expectations (triangle inequality) and applying the Tonelli theorem to the nonnegative product-measurable dominating integrand gives

which is finite because the moduli are bounded (part (a)) and $\int_{[0,T]}\mathbb{E}[m_t]dt<\infty$, $\mathbb{E}[|\mathfrak{s}_T|^2]<\infty$ (Step 3). This completes the proof of (c).