Proof of Martingale Decomposition of the Empirical State Measure and the Observation Process

Throughout, fix a solution and work with its occupation indicators $\eta^{i,\gamma}_t$, counters $N^{i,\sigma\gamma}_t$, $\tilde{N}^{i,\upsilon}_t$, consumed clock times $\mathcal{T}^{i,\sigma\gamma}_t$, $\tilde{\mathcal{T}}^{i,\upsilon}_t$, regular event $\Omega_0$, and system filtration, all in the sense of Solution of the Controlled N-Agent Dynamics; pathwise identities below are asserted on $\Omega_0$, where conditions 1--6 of that definition hold. Write $M^a_t=N^a_t-\mathcal{T}^a_t$ for the compensated counters, indexed by clock labels $a$ as in that lemma.

Step 1: part (a). Each path $s\mapsto\eta^{i,\gamma}_s$ is piecewise constant with finitely many pieces by condition 1 of Solution of the Controlled N-Agent Dynamics, hence measurable on $[0,T]$ with the trace Borel $\sigma$-algebra, and therefore so is each component $s\mapsto\Sigma^\gamma_s$, a finite sum of indicators divided by $N$; measurability of finite sums follows from Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable applied with $g$ the (continuous) sum of coordinates and $f$ the vector of indicator paths. Similarly, by condition 5 of Solution of the Controlled N-Agent Dynamics, on each of the finitely many intervals between observation events (finitely many: by condition 3 the observation total agrees on $[0,T]$ with a counting path, which is nondecreasing, integer-valued and increases by exactly $1$ at each jump, so the number of its jumps in $[0,T]$ is the finite value $\tilde{c}_T$) the control path $s\mapsto\alpha_s$ agrees with a section $s\mapsto h^j_k(s,\tau^\circ,v^\circ)$ of a policy function with the record entries frozen; such a section is measurable because $s\mapsto(s,\tau^\circ)$ maps $[0,T]$ into $[0,T]\times R_k(T)$ and pulls each relatively open set back to a relatively open subset of $[0,T]$, hence pulls the generated $\sigma$-algebra into the trace Borel $\sigma$-algebra. So each component path $s\mapsto\alpha^j_s$ is measurable on $[0,T]$ (a finite patching of measurable functions on subintervals). Now each of the functions $(\Sigma,\alpha)\mapsto b^\gamma(\Sigma,\alpha)$, $\Sigma\mapsto\tilde{b}^\upsilon(\Sigma)$, and $(\Sigma,\alpha)\mapsto\Theta^{\gamma\delta}(\Sigma,\alpha)$ is a finite sum of products of coordinate maps and members of the transition-rate family or observation-rate family, hence sequentially continuous on $\Delta^l\times\mathcal{A}$ (respectively $\Delta^l$) by the continuity conditions of those definitions together with the fact that limits of products of convergent real sequences are the products of the limits; therefore Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable applies to the measurable path $s\mapsto(\Sigma_s,\alpha_s)$ and yields the asserted path measurability of $s\mapsto b^\gamma(\Sigma_s,\alpha_s)$, $s\mapsto\tilde{b}^\upsilon(\Sigma_s)$, $s\mapsto\Theta^{\gamma\delta}(\Sigma_s,\alpha_s)$ on $\Omega_0$.

For the bounds, recall from Transition-Rate Family that $0\le\beta\le B$, from Observation-Rate Family that $0\le\tilde{\beta}\le\tilde{B}$, and from Probability Simplex that $\Sigma^\gamma_s\ge0$ and $\sum_\gamma\Sigma^\gamma_s=1$. From the formula in Aggregate State Drift, $|b^\gamma|\le B\sum_{\sigma\neq\gamma}\Sigma^\sigma+(l-1)B\Sigma^\gamma\le B+(l-1)B\le 2(l-1)B$ since $l\ge2$. From Aggregate Fluctuation Covariance, the diagonal entries satisfy $|\Theta^{\gamma\gamma}|\le B\sum_{\sigma\neq\gamma}\Sigma^\sigma+(l-1)B\Sigma^\gamma\le 2(l-1)B$ likewise, and the off-diagonal entries satisfy $|\Theta^{\gamma\delta}|\le B(\Sigma^\gamma+\Sigma^\delta)\le 2B\le 2(l-1)B$. From Aggregate Observation Drift, $|\tilde{b}^\upsilon|\le\tilde{B}\sum_\sigma\Sigma^\sigma=\tilde{B}$. Bounded measurable paths are Lebesgue integrable on $[0,t]$ by Restricted Lebesgue Measure and Integral Toolkit on a Compact Interval, so all the integrals exist. This proves (a).

Step 2: the aggregate martingales are combinations of compensated counters. Fix $\gamma$ and work on $\Omega_0$. Summing condition 6 of Solution of the Controlled N-Agent Dynamics over $i$ and dividing by $N$,

On the other hand, for fixed $\sigma\neq\gamma'$, the identity $\frac{1}{N}\sum_i\eta^{i,\sigma}_s\,\beta(\sigma,\gamma',\Sigma_s,\alpha_s)=\Sigma^\sigma_s\,\beta(\sigma,\gamma',\Sigma_s,\alpha_s)$ holds pointwise in $s$ on $\Omega_0$, so by linearity of the Lebesgue integral and the definition of the consumed clock times in Solution of the Controlled N-Agent Dynamics,

Combining with the formula for $b^\gamma$ in Aggregate State Drift and subtracting, on $\Omega_0$,

that is, $M^\gamma_t=\frac{1}{N}\sum_a c^\gamma_a M^a_t$ where the sum runs over transition clock labels $a=(i,\sigma\gamma')$ and $c^\gamma_{(i,\sigma\gamma')}=\mathbf{1}_{\{\gamma'=\gamma\}}-\mathbf{1}_{\{\sigma=\gamma\}}\in\{-1,0,1\}$ (we write $\mathbf{1}_{\{\cdot\}}$ for the indicator equal to $1$ when the subscripted condition holds and $0$ otherwise). Entirely analogously, condition 4 of Solution of the Controlled N-Agent Dynamics, the identity $\frac{1}{N}\sum_i\tilde{\beta}(\sigma^i_s,\upsilon,\Sigma_s)=\sum_\sigma\Sigma^\sigma_s\tilde{\beta}(\sigma,\upsilon,\Sigma_s)=\tilde{b}^\upsilon(\Sigma_s)$ (valid on $\Omega_0$ since $\tilde{\beta}(\sigma^i_s,\upsilon,\Sigma_s)=\sum_\sigma\eta^{i,\sigma}_s\tilde{\beta}(\sigma,\upsilon,\Sigma_s)$), and Aggregate Observation Drift give, on $\Omega_0$,

where $M^{(i,\upsilon)}_t=\tilde{N}^{i,\upsilon}_t-\tilde{\mathcal{T}}^{i,\upsilon}_t$ is the compensated counter of the observation clock label $(i,\upsilon)$.

Step 3: part (b). By part (a) of the compensated counter lemma, each $M^a$ is a square-integrable martingale with respect to the system filtration with $M^a_0=0$, and a finite linear combination of square-integrable martingales with real coefficients is again one: adaptedness and square-integrability are preserved under finite linear combinations (for square-integrability use the triangle inequality for the mean-square norm), and the martingale property in the averaged form of Square-Integrable Martingale, Submartingale, and Supermartingale is linear in the process. The Step 2 identities hold at every point of $\Omega_0$, so $M^\gamma_t$ (and likewise $\tilde{M}^\upsilon_t$) agrees on $\Omega_0$, hence almost surely, with the corresponding combination $Z_t=\frac1N\sum_a c^\gamma_aM^a_t$. Square-integrability and the averaged martingale identity transfer under almost sure equality, since expectations of the involved products are unchanged.

Adaptedness needs more care, because $\mathcal{F}$ is not assumed complete and a subset of an event of probability zero need not itself be an event; we therefore first prove that $M^\gamma_t$ is $\mathcal{F}$-measurable. The map

is a finite linear combination of the very maps that condition 2 of Solution of the Controlled N-Agent Dynamics requires to be jointly measurable. The displayed identity holds at every point of $\Omega$: off $\Omega_0$ both sides vanish, while on $\Omega_0$ it is exactly the computation of Step 2, where $\mathbf{1}_{\Omega_0}=1$. Hence the map is jointly measurable, and it is bounded by part (a). Applying the Tonelli theorem to its positive and negative parts, on the product of $[0,t]$ carrying Lebesgue measure with the probability space, shows that $\omega\mapsto\int_{[0,t]}\mathbf{1}_{\Omega_0}b^\gamma(\Sigma_s,\alpha_s)\,ds$ is $\mathcal{F}$-measurable. Since $\Sigma^\gamma_t$ and $\Sigma^\gamma_0$ are random variables, $M^\gamma_t$ is $\mathcal{F}$-measurable. The same argument, applied to the second integrand of condition 2 together with the identity $\frac{1}{N}\sum_i\mathbf{1}_{\Omega_0}\tilde{\beta}(\sigma^i_s,\upsilon,\Sigma_s)=\mathbf{1}_{\Omega_0}\tilde{b}^\upsilon(\Sigma_s)$, which is the corresponding identity of Step 2 multiplied by $\mathbf{1}_{\Omega_0}$ and so valid at every point of $\Omega$, shows that $\tilde{M}^\upsilon_t$ is $\mathcal{F}$-measurable.

Now fix a Borel set $S$. Since $M^\gamma_t=Z_t$ at every point of $\Omega_0$,

Here $\{Z_t\in S\}\in\mathcal{F}^{\mathrm{sys}}_t$, because $Z_t$ is a finite linear combination of the variables $M^a_t$, which are adapted to the system filtration by part (a) of Compensated Counters of the Controlled N-Agent Dynamics are Square-Integrable Martingales. The second set is an event, by the $\mathcal{F}$-measurability just established, and it is contained in $\Omega\setminus\Omega_0$, an event of probability zero, so it is itself an event of probability zero. By Solution of the Controlled N-Agent Dynamics the system filtration contains every event of $\mathcal{F}$ of probability zero; in particular it contains $\Omega\setminus\Omega_0$, hence also $\Omega_0$, and it contains the second set. Both sets on the right therefore lie in $\mathcal{F}^{\mathrm{sys}}_t$, and hence so does their union, giving $\{M^\gamma_t\in S\}\in\mathcal{F}^{\mathrm{sys}}_t$. The same argument applies to $\tilde{M}^\upsilon_t$. Hence each $M^\gamma$ and each $\tilde{M}^\upsilon$ is a square-integrable martingale. The initial values are as asserted in (b). On the one hand $M^\gamma_0=\Sigma^\gamma_0-\Sigma^\gamma_0-0=0$ at every point of $\Omega$, the integral over $[0,0]$ vanishing because $\{0\}$ is a Lebesgue null set. On the other hand $\tilde{M}^\upsilon_0=\Upsilon^\upsilon_0$, which vanishes at every $\omega\in\Omega_0$ by conditions 3 and 4 (the counters vanish at $t=0$ because the consumed clock times do and clock paths are counting paths), hence almost surely; off $\Omega_0$ no claim is made about it. This proves (b).

Step 4: part (c). Fix $0\le r\le t\le T$ and $D\in\mathcal{F}^{\mathrm{sys}}_r$. All the products below are integrable by the integrability assertion of the compensated counter lemma and finite summation, and almost sure equality (Step 3) lets us compute with $Z_t=\frac1N\sum_a c^\gamma_aM^a_t$ in place of $M^\gamma_t$. By Step 2 and bilinearity,

Applying part (b) of Compensated Counters of the Controlled N-Agent Dynamics are Square-Integrable Martingales to every pair $(a,b)$ and re-assembling the $r$-terms by the same bilinearity,

It remains to identify the last sum. For a transition clock label $a=(i,\sigma\gamma')$ we have, as in Step 2, $\sum_i (\mathcal{T}^{i,\sigma\gamma'}_t-\mathcal{T}^{i,\sigma\gamma'}_r)=N\int_{[r,t]}\mathbf{1}_{\Omega_0}\,\Sigma^\sigma_s\beta(\sigma,\gamma',\Sigma_s,\alpha_s)\,ds$ at every point of $\Omega$ (additivity of the Lebesgue integral over $[0,r]$ and $[r,t]$ from Restricted Lebesgue Measure and Integral Toolkit on a Compact Interval). Hence, at every point of $\Omega$,

For $\gamma=\delta$, the summand weight $(\mathbf{1}_{\{\gamma'=\gamma\}}-\mathbf{1}_{\{\sigma=\gamma\}})^2$ equals $1$ exactly when $\gamma'=\gamma$ or $\sigma=\gamma$ (never both, as $\sigma\neq\gamma'$) and $0$ otherwise, so the inner sum equals $\sum_{\sigma\neq\gamma}\Sigma^\sigma\beta(\sigma,\gamma,\Sigma_s,\alpha_s)+\sum_{\gamma'\neq\gamma}\Sigma^\gamma\beta(\gamma,\gamma',\Sigma_s,\alpha_s)=\Theta^{\gamma\gamma}(\Sigma_s,\alpha_s)$ by Aggregate Fluctuation Covariance. For $\gamma\neq\delta$, the weight is nonzero only for $(\sigma,\gamma')=(\gamma,\delta)$, where it is $(\mathbf{1}_{\{\delta=\gamma\}}-\mathbf{1}_{\{\gamma=\gamma\}})(\mathbf{1}_{\{\delta=\delta\}}-\mathbf{1}_{\{\gamma=\delta\}})=(-1)(1)=-1$, and for $(\sigma,\gamma')=(\delta,\gamma)$, where it is likewise $-1$; all other pairs give $0$. The inner sum is then $-\Sigma^\gamma\beta(\gamma,\delta,\Sigma_s,\alpha_s)-\Sigma^\delta\beta(\delta,\gamma,\Sigma_s,\alpha_s)=\Theta^{\gamma\delta}(\Sigma_s,\alpha_s)$ by Aggregate Fluctuation Covariance. Taking expectations of this pathwise identity and combining with the previous display proves the first identity of (c).

For the observation covariations, Step 2 gives $\tilde{M}^\upsilon=\frac{1}{N}\sum_i M^{(i,\upsilon)}$ almost surely, and part (b) of Compensated Counters of the Controlled N-Agent Dynamics are Square-Integrable Martingales leaves only the diagonal pairs $(i,\upsilon)=(i',\upsilon')$, which force $\upsilon=\upsilon'$: when $\upsilon\neq\upsilon'$ no diagonal pairs occur and the $r$-identity holds with nothing added, while for $\upsilon=\upsilon'$ the added term is $\frac{1}{N^2}\sum_i\mathbb{E}[\mathbf{1}_D(\tilde{\mathcal{T}}^{i,\upsilon}_t-\tilde{\mathcal{T}}^{i,\upsilon}_r)]=\frac{1}{N}\mathbb{E}\big[\mathbf{1}_D\int_{[r,t]}\mathbf{1}_{\Omega_0}\,\tilde{b}^\upsilon(\Sigma_s)\,ds\big]$, using $\frac{1}{N}\sum_i\mathbf{1}_{\Omega_0}\tilde{\beta}(\sigma^i_s,\upsilon,\Sigma_s)=\mathbf{1}_{\Omega_0}\tilde{b}^\upsilon(\Sigma_s)$ at every point of $\Omega$ as in Step 2 and integral additivity from Restricted Lebesgue Measure and Integral Toolkit on a Compact Interval. Finally, a state martingale $M^\gamma$ and an observation martingale $\tilde{M}^\upsilon$ are combinations over disjoint sets of clock labels (transition clocks versus observation clocks), so part (b) of Compensated Counters of the Controlled N-Agent Dynamics are Square-Integrable Martingales contributes no diagonal terms at all and the mixed identity holds with nothing added. This proves (c). $\blacksquare$