Proof of Comparison of the N-Agent System with the Mean-Field Flow and Cost along the Realized Control

Write $\mathcal{G}=\mathcal{B}_{[0,T]}\otimes\mathcal{F}$ for the product $\sigma$-algebra on $[0,T]\times\Omega$, and $\lambda=\lambda_{[0,T]}$. Real-valued maps are called measurable when they are measurable for the relevant $\sigma$-algebra and the Borel $\sigma$-algebra of the real line. We use repeatedly that sums, differences, products, absolute values and maxima of finitely many measurable real-valued maps are measurable, and that a product of a measurable map with the indicator of a measurable set is measurable: in each case the map in question is a sequentially continuous function of finitely many measurable real-valued maps, so Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable applies, the indicator of a measurable set being measurable because its preimages are $\emptyset$, that set, its complement, or the whole space. For $\omega\in\Omega$ write $S^{\omega}=S\bigl(\Sigma_{0}(\omega),\hat{\alpha}(\omega)\bigr)$, which is defined since $\Sigma_{0}(\omega)\in\Delta^{l}$ and $\hat{\alpha}(\omega)\in\mathcal{U}_{\mathcal{A}}$.

Step 0: joint measurability of the empirical state measure on $\Omega_{*}$. We show that every component of the map $\Sigma^{*}$ of claim 2 is $\mathcal{G}$-measurable. For a natural number $n$ let $D_{n}=\{kT2^{-n}:k\in\{0,1,\dots,2^{n}\}\}$ and, for $t\in[0,T]$, let $t_{n}$ be the least element of $D_{n}$ with $t_{n}\ge t$. Define $\Sigma^{*,n}_{t}(\omega)=\Sigma_{t_{n}}(\omega)$ for $\omega\in\Omega_{*}$ and $\Sigma^{*,n}_{t}(\omega)=e$ for $\omega\notin\Omega_{*}$. For each $\gamma$ the map $(t,\omega)\mapsto(\Sigma^{*,n}_{t})^{\gamma}(\omega)$ is a finite sum, over the finitely many elements $s\in D_{n}$, of the product of the indicator of $\{t\in[0,T]:t_{n}=s\}\times\Omega$, a set in $\mathcal{G}$ because $\{t:t_{n}=s\}$ is an interval, with the random variable $\Sigma^{\gamma}_{s}\mathbf{1}_{\Omega_{*}}$ ($\Sigma^{\gamma}_{s}$ being $\mathcal{F}$-measurable by claim (iv) of the $N$-agent existence and uniqueness theorem), plus $e^{\gamma}\mathbf{1}_{[0,T]\times(\Omega\setminus\Omega_{*})}$; hence it is $\mathcal{G}$-measurable.

Fix $(t,\omega)$. If $\omega\notin\Omega_{*}$ then $(\Sigma^{*,n}_{t})^{\gamma}(\omega)=e^{\gamma}=(\Sigma^{*}_{t})^{\gamma}(\omega)$ for every $n$. If $\omega\in\Omega_{*}$ and $t=T$ then $t_{n}=T$ for every $n$ and again the values agree. If $\omega\in\Omega_{*}$ and $t<T$ then $t\le t_{n}\le t+T2^{-n}$, so, given $\varepsilon>0$, the right-continuity of the path $s\mapsto\Sigma_{s}(\omega)$ at $t$, which holds by the choice of $\Omega_{*}$ in the martingale bound, provides $\eta>0$ with $|\Sigma_{s}(\omega)-\Sigma_{t}(\omega)|\le\varepsilon$ for $t\le s\le\min(t+\eta,T)$, and $t_{n}$ lies in that range as soon as $T2^{-n}\le\eta$, which happens for all large $n$ by The Archimedean Property of the Real Numbers. So in every case the sequence $\bigl((\Sigma^{*,n}_{t})^{\gamma}(\omega)\bigr)_{n}$ converges to $(\Sigma^{*}_{t})^{\gamma}(\omega)$. All these maps are bounded in absolute value by $1$, by claim 1 of the compactness lemma and claim 4 of Elementary Properties of the Euclidean Norm on $\mathbb{R}^n$. Claim 2 of Measurability of Countable Suprema, Bounded Pointwise Limits, Monotone Functions, and Continuous Functions now gives the $\mathcal{G}$-measurability of each component of $\Sigma^{*}$. Since $\Sigma_{t}(\omega)\in\Delta^{l}$ by claim (vii)(a) of Existence, Uniqueness, and Regularity for the Controlled N-Agent Dynamics and $e\in\Delta^{l}$, all values of $\Sigma^{*}$ lie in $\Delta^{l}$.

Claim 1. Fix $\omega\in\Omega_{*}$ and abbreviate $\Sigma_{t}=\Sigma_{t}(\omega)$, $S=S^{\omega}$, and let $u$ be the path $t\mapsto\hat{\alpha}(t,\omega)$, an admissible representative of $\hat{\alpha}(\omega)$ by claim 3 of the realized-control lemma. By claim 2 of the flow stability lemma, $S$ is the map furnished by claim 1 of the generalized mean-field existence and uniqueness theorem for the initial value $\Sigma_{0}(\omega)$ and the control $u$; in particular $S_{t}\in\Delta^{l}$ for every $t$, the path $S$ satisfies $|S_{t}-S_{r}|\le K_{b}|t-r|$ with the Lipschitz constant $K_{b}$ furnished by claim 1 of that generalized mean-field existence and uniqueness theorem, and

$$S_{t}=\Sigma_{0}(\omega)+\int_{[0,t]}\hat{b}\bigl(S_{s},u(s)\bigr)\,ds\qquad(t\in[0,T]),$$

where $\hat{b}$ is the projected drift of the lemma on affine-controlled data, since that existence theorem is stated with the projected drift. Because $S_{s}\in\Delta^{l}$ for every $s$, claim 6 of that lemma gives $\hat{b}(S_{s},u(s))=b(S_{s},u(s))$, so the integrand may equally be written with the aggregate state drift:

$$S_{t}=\Sigma_{0}(\omega)+\int_{[0,t]}b\bigl(S_{s},u(s)\bigr)\,ds\qquad(t\in[0,T]).$$

By claim (b) of the martingale decomposition,

$$\Sigma_{t}=\Sigma_{0}(\omega)+\int_{[0,t]}b\bigl(\Sigma_{s},\alpha_{s}(\omega)\bigr)\,ds+M_{t}(\omega)\qquad(t\in[0,T]),$$

and $\alpha_{s}(\omega)=u(s)$ for every $s\in[0,T]$, because $\omega\in\Omega_{*}\subseteq\Omega_{0}$ and claim 2 of the realized-control lemma gives the agreement of $\hat{\alpha}$ with $\alpha$ on $\Omega_{0}$. Subtracting the two displays and using the triangle inequality for the Euclidean norm, claim 6 of Elementary Properties of the Euclidean Norm on $\mathbb{R}^n$, then Norm Bound for a Vector-Valued Lebesgue Integral over a Compact Interval to bound the norm of the integral by the integral of the norm, then the state-Lipschitz bound for $b$ recalled in the hypotheses, and finally $|M_{t}(\omega)|\le\overline{M}(\omega)$, we obtain

$$|\Sigma_{t}-S_{t}|\le\overline{M}(\omega)+\Lambda_{b}\int_{[0,t]}|\Sigma_{s}-S_{s}|\,ds\qquad(t\in[0,T]).$$

Put $\phi(t)=|\Sigma_{t}-S_{t}|$. Both $\Sigma_{t}$ and $S_{t}$ lie in $\Delta^{l}$, so $\phi(t)\le2$ by claim 1 of the compactness lemma and the triangle inequality: $\phi$ is bounded. It is also measurable on $[0,T]$: every component of $t\mapsto\Sigma^{*}_{t}(\omega)$ is measurable, being a section of a $\mathcal{G}$-measurable map by the statement on sections in Tonelli and Fubini Theorems applied on the product of the finite measure spaces $([0,T],\mathcal{B}_{[0,T]},\lambda)$ and $(\Omega,\mathcal{F},P)$, and $\Sigma^{*}_{t}(\omega)=\Sigma_{t}$ here; the path $S$ is measurable in each component, being Lipschitz, hence continuous, hence measurable by claim 4 of Measurability of Countable Suprema, Bounded Pointwise Limits, Monotone Functions, and Continuous Functions; and $(x,y)\mapsto|x-y|$ is sequentially continuous on $\mathbb{R}^{l}\times\mathbb{R}^{l}$, so Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable applies. Therefore Gronwall's Lemma for Bounded Measurable Functions, with $a=\overline{M}(\omega)$ and $c=\Lambda_{b}$, yields $\phi(t)\le\overline{M}(\omega)e^{\Lambda_{b}t}$ for every $t\in[0,T]$, and hence $\phi(t)\le e^{\Lambda_{b}T}\overline{M}(\omega)$, since $\overline{M}(\omega)\ge0$ and $e^{\Lambda_{b}t}\le e^{\Lambda_{b}T}$.

Claim 2. By claim 3 of the boundedness, lower-semicontinuity and attainment theorem, $F$ is lower semicontinuous on $X$ for $d_{X}$, hence measurable with respect to the Borel $\sigma$-algebra of $(X,d_{X})$ and $\mathcal{B}(\mathbb{R})$ by claim 5 of Borel Measurability and Bounded Integration on a Metric Space. The map $\omega\mapsto(\Sigma_{0}(\omega),\hat{\alpha}(\omega))$ is a random element of $(X,d_{X})$ by claim 5 of the realized-control lemma, so the composite $\omega\mapsto F(\Sigma_{0}(\omega),\hat{\alpha}(\omega))$ is a random variable by claim 4 of Borel Measurability and Bounded Integration on a Metric Space. Its bound by $C_{F}$ is claim 1 of that theorem.

Next, put $g(t,\omega)=L\bigl(\Sigma^{*}_{t}(\omega),\hat{\alpha}(t,\omega)\bigr)$. The components of $(t,\omega)\mapsto\bigl(\Sigma^{*}_{t}(\omega),\hat{\alpha}(t,\omega)\bigr)$ are $\mathcal{G}$-measurable by Step 0 and claim 2 of the realized-control lemma, this map takes values in the nonempty subset $\Delta^{l}\times\mathbb{R}^{m}$ of $\mathbb{R}^{l+m}$, and $L$ is sequentially continuous there by condition 1 of Population Cost Data; so $g$ is $\mathcal{G}$-measurable by Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable. Since $\Sigma^{*}_{t}(\omega)\in\Delta^{l}$ and $\hat{\alpha}(t,\omega)\in\mathcal{A}$, claim 1 of the lemma on cost data over a compact control set gives $|g|\le C$. Writing $g^{+}=\max(g,0)$ and $g^{-}=\max(-g,0)$, the Tonelli statement of Tonelli and Fubini Theorems makes the map $\omega\mapsto\int_{[0,T]}g^{\pm}(t,\omega)\,d\lambda(t)$ measurable with respect to $\mathcal{F}$, with values in $[0,\infty]$, and both are at most $CT$, hence real; their difference is $\int_{[0,T]}g(t,\omega)\,d\lambda(t)$, which is therefore a random variable bounded in absolute value by $CT$. Also $\omega\mapsto G(\Sigma^{*}_{T}(\omega))$ is a random variable by Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable, bounded by $C$. Hence $W$ is a random variable with $|W|\le C(T+1)$.

Finally let $\Xi(\omega)=\int_{[0,T]}L(\Sigma_{t},\alpha_{t})\,dt+G(\Sigma_{T})$ be the extended-real-valued map whose expectation is $J^{N}[h]$ by The N-Agent Cost Functional. By claim (v) of the $N$-agent existence and uniqueness theorem, $\Xi$ is measurable as an extended-real-valued map, is bounded below by $-C_{L}T-C_{G}$, where $C_{L}$ and $C_{G}$ are the lower bounds of clause 2 of Population Cost Data, and $\mathbb{E}[\Xi]$ is the limit of the expectations of the truncations $\Xi_{n}=\min(\Xi,n)$. For $\omega\in\Omega_{*}$ we have $\Sigma^{*}_{t}(\omega)=\Sigma_{t}(\omega)$ and $\alpha_{t}(\omega)=\hat{\alpha}(t,\omega)$ for every $t$; hence for such $\omega$ the path $t\mapsto L(\Sigma_{t}(\omega),\alpha_{t}(\omega))$ coincides with the section $t\mapsto g(t,\omega)$, which is measurable by the statement on sections in Tonelli and Fubini Theorems. So $\omega$ does not lie in the exceptional event of clause (v) on which the integral is set to $0$, and the integral defining $\Xi(\omega)$ is the Lebesgue integral of that path. Therefore $\Xi(\omega)=W(\omega)$ and $|\Xi(\omega)|\le C(T+1)$. Therefore, for every natural number $n\ge C(T+1)$, the random variables $\Xi_{n}$ and $W$ agree at every point of $\Omega_{*}$, and both are bounded on all of $\Omega$, the first between $-C_{L}T-C_{G}$ and $n$. Claim 2 of Almost Sure Inequalities Between Bounded Random Variables Pass to Expectations, applied with the event $\Omega_{*}$ of probability $1$, gives $\mathbb{E}[\Xi_{n}]=\mathbb{E}[W]$ for every such $n$. Consequently $J^{N}[h]=\mathbb{E}[\Xi]=\mathbb{E}[W]$, a real number.

Claim 3. Let $\varepsilon>0$ and put $\varepsilon'=\varepsilon/\bigl(2(T+1)\bigr)>0$. By claim 2 of the lemma on cost data over a compact control set there is a real $\delta>0$, depending only on $L$, $G$, $\Delta^{l}$, $\mathcal{A}$, $T$ and $\varepsilon$, such that

$$|L(\Sigma,a)-L(\Sigma',a)|\le\varepsilon'\quad\text{and}\quad|G(\Sigma)-G(\Sigma')|\le\varepsilon'$$

whenever $\Sigma,\Sigma'\in\Delta^{l}$ satisfy $|\Sigma-\Sigma'|\le\delta$ and $a\in\mathcal{A}$. Put $\eta=\delta e^{-\Lambda_{b}T}>0$ and let $B_{N}=\{\overline{M}>\eta\}$, an event since $\overline{M}$ is a random variable.

For every $\omega\in\Omega$, claim 2 of the attainment theorem shows that $S^{\omega}$ is an admissible state path and that $F(\Sigma_{0}(\omega),\hat{\alpha}(\omega))=\Phi_{S^{\omega}}(\hat{\alpha}(\omega))+G(S^{\omega}_{T})$; since the path $t\mapsto\hat{\alpha}(t,\omega)$ is everywhere $\mathcal{A}$-valued, claim 1 of the running-cost lemma evaluates the first term as an integral, so

$$F\bigl(\Sigma_{0}(\omega),\hat{\alpha}(\omega)\bigr)=\int_{[0,T]}L\bigl(S^{\omega}_{t},\hat{\alpha}(t,\omega)\bigr)\,d\lambda(t)+G\bigl(S^{\omega}_{T}\bigr).$$

Let $\omega\in\Omega_{*}$ with $\omega\notin B_{N}$. Then $\overline{M}(\omega)\le\eta$, so claim 1 gives $|\Sigma_{t}(\omega)-S^{\omega}_{t}|\le e^{\Lambda_{b}T}\eta=\delta$ for every $t\in[0,T]$; as $\Sigma^{*}_{t}(\omega)=\Sigma_{t}(\omega)$ and both points lie in $\Delta^{l}$, the choice of $\delta$ and the monotonicity of the integral, claim 1 of Linearity and Monotonicity of the Lebesgue Integral, give

$$\bigl|W(\omega)-F\bigl(\Sigma_{0}(\omega),\hat{\alpha}(\omega)\bigr)\bigr|\le\int_{[0,T]}\bigl|L(\Sigma_{t},\hat{\alpha}(t,\omega))-L(S^{\omega}_{t},\hat{\alpha}(t,\omega))\bigr|\,d\lambda+\bigl|G(\Sigma_{T})-G(S^{\omega}_{T})\bigr|\le\varepsilon'T+\varepsilon'\le\frac{\varepsilon}{2}.$$

For $\omega\in\Omega_{*}\cap B_{N}$ we use instead the crude bound $|W-F|\le C(T+1)+C_{F}=:C_{2}$, from claim 2. Hence, on all of $\Omega_{*}$,

$$\bigl|W-F(\Sigma_{0},\hat{\alpha})\bigr|\le\frac{\varepsilon}{2}+C_{2}\,\mathbf{1}_{B_{N}} .$$

Both sides are random variables bounded on all of $\Omega$, so claim 1 of Almost Sure Inequalities Between Bounded Random Variables Pass to Expectations, applied with the event $\Omega_{*}$, yields

$$\mathbb{E}\bigl[\bigl|W-F(\Sigma_{0},\hat{\alpha})\bigr|\bigr]\le\frac{\varepsilon}{2}+C_{2}\,P(B_{N}).$$

Since $\overline{M}\ge0$ and $\eta>0$, the event $B_{N}$ is contained in $\{\overline{M}^{2}\ge\eta^{2}\}$, so Markov's inequality, Markov's and Chebyshev's Inequalities, applied to the nonnegative random variable $\overline{M}^{2}$, together with claim 2 of Basic Properties of a Measure and the mean-square bound of the martingale bound, gives

$$P(B_{N})\le\frac{\mathbb{E}\bigl[\overline{M}^{2}\bigr]}{\eta^{2}}\le\frac{8l(l-1)BT}{N\eta^{2}} .$$

By The Archimedean Property of the Real Numbers choose a natural number $N_{0}$ with

$$N_{0}\ \ge\ \frac{2\,C_{2}\,8\,l\,(l-1)\,B\,T}{\varepsilon\,\eta^{2}} .$$

Then $C_{2}P(B_{N})\le\varepsilon/2$ for every $N\ge N_{0}$, and therefore, using $J^{N}[h]=\mathbb{E}[W]$ from claim 2, the linearity of the expectation and the bound $|\mathbb{E}[V]|\le\mathbb{E}[|V|]$ for a bounded random variable $V$, both from Linearity and Monotonicity of the Lebesgue Integral,

$$\Bigl|J^{N}[h]-\mathbb{E}\bigl[F(\Sigma_{0},\hat{\alpha})\bigr]\Bigr|=\Bigl|\mathbb{E}\bigl[W-F(\Sigma_{0},\hat{\alpha})\bigr]\Bigr|\le\mathbb{E}\bigl[\bigl|W-F(\Sigma_{0},\hat{\alpha})\bigr|\bigr]\le\varepsilon .$$

It remains to note the dependence of $N_{0}$. The constants $C$ and $C_{F}$, hence $C_{2}$, depend only on $L$, $G$, $\Delta^{l}$, $\mathcal{A}$ and $T$; the number $\delta$ depends only on $L$, $G$, $\Delta^{l}$, $\mathcal{A}$, $T$ and $\varepsilon$, and $\eta$ only on those together with $\Lambda_{b}$ and $T$; and the displayed lower bound for $N_{0}$ involves besides these only $l$, $B$, $T$ and $\varepsilon$. None of them refers to $N$, to the driving system, to the policy or to the solution, so a single $N_{0}$ serves for all of them simultaneously. $\blacksquare$