Solution of the Controlled N-Agent Dynamics

Let $N$, $l$, $\tilde{l}$, $m$ be natural numbers with $N\ge1$, $l\ge2$, $\tilde{l}\ge1$, $m\ge1$. Fix a transition-rate family $\beta$ on $l$ states with control dimension $m$ and rate bound $B$, an observation-rate family $\tilde{\beta}$ on $l$ states with $\tilde{l}$ observation channels and rate bound $\tilde{B}$, a real number $T>0$, an $N$-agent driving system $(\Omega,\mathcal{F},P)$ with initial states $\varsigma^i_0$, transition clocks $Y^{i,\sigma\gamma}$, and observation clocks $\tilde{Y}^{i,\upsilon}$, and an observation-driven control policy $h=(h_k)_{k\ge0}$ with horizon $T$, control dimension $m$, and $\tilde{l}$ channels.

A solution of the controlled $N$-agent dynamics on $[0,T]$ consists of families of random variables indexed by $t\in[0,T]$ on $(\Omega,\mathcal{F},P)$ (stochastic processes with time restricted to $[0,T]$), namely state processes $\sigma^i=(\sigma^i_t)$ taking values in $\{1,\dots,l\}$ for $i\in\{1,\dots,N\}$, observation processes $\Upsilon^\upsilon=(\Upsilon^\upsilon_t)$ taking real values for $\upsilon\in\{1,\dots,\tilde{l}\}$, and a control process $\alpha=(\alpha_t)$ taking values in $\mathbb{R}^m$ (each component $\alpha^j$ a real-valued process), together with an event $\Omega_0\in\mathcal{F}$ with $P(\Omega_0)=1$, called the regular event, such that conditions 1--6 below hold at every $\omega\in\Omega_0$. The conditions are a joint requirement on the whole collection: condition 2 refers to the control appearing in condition 5, and condition 5 to the counters of condition 3.

Derived notation: the occupation indicators are $\eta^{i,\gamma}_t=1$ if $\sigma^i_t=\gamma$ and $\eta^{i,\gamma}_t=0$ otherwise; the empirical state measure is $\Sigma_t=(\Sigma^1_t,\dots,\Sigma^l_t)$ with $\Sigma^\gamma_t=\frac{1}{N}\sum_{i=1}^N\eta^{i,\gamma}_t$, which lies in the probability simplex $\Delta^l$ because each agent occupies exactly one state.

1. (State regularity.) For each $i$: $\sigma^i_0=\varsigma^i_0$, and the path $t\mapsto\sigma^i_t$ is piecewise constant and right-continuous: there are a count $K^{(i)}$, either zero or a natural number, and times $0<t^{(i)}_1<\dots<t^{(i)}_{K^{(i)}}\le T$ such that $t\mapsto\sigma^i_t$ is constant on each of the intervals $[0,t^{(i)}_1),\dots,[t^{(i)}_{K^{(i)}},T]$, with the convention that for $K^{(i)}=0$ the path is constant on all of $[0,T]$.

2. (Joint measurability and time changes.) For all $i$, all ordered pairs $(\sigma,\gamma)$ with $\sigma\neq\gamma$, and all $\upsilon$, the maps

are measurable with respect to the product $\sigma$-algebra of the trace Borel $\sigma$-algebra on $[0,T]$ and $\mathcal{F}$, where $\mathbf{1}_{\Omega_0}$ is the function equal to $1$ on $\Omega_0$ and $0$ off $\Omega_0$. These maps take values in $[0,B]$ and $[0,\tilde{B}]$ respectively, so for every $\omega\in\Omega$ the consumed clock times

exist as Lebesgue integrals over the compact interval $[0,t]$ applied to the sections in $s$ (sections of jointly measurable maps are measurable, as in the Tonelli theorem), satisfy $0\le A^{i,\sigma\gamma}_t\le Bt$ and $0\le\tilde{A}^{i,\upsilon}_t\le\tilde{B}t$, vanish identically off $\Omega_0$, and are $\mathcal{F}$-measurable in $\omega$ for each fixed $t$ by the Tonelli theorem. Thus the consumed clock times are defined on all of $\Omega$.

3. (Counting structure.) Define the transition counters and observation counters on all of $\Omega$ by

off $\Omega_0$ these vanish, since the consumed clock times vanish there and every clock path is a counting path, and each of them is a random variable, being the evaluation of a right-continuous path at a measurable time (a pointwise limit, over rational levels decreasing to the consumed clock time, of the clock variables). It is required that each of the maps $t\mapsto N^{i,\sigma\gamma}_t$, each of the maps $t\mapsto\tilde{N}^{i,\upsilon}_t$, the observation total $t\mapsto\tilde{c}_t=\sum_{i=1}^N\sum_{\upsilon=1}^{\tilde{l}}\tilde{N}^{i,\upsilon}_t$, and the grand total obtained by adding to $\tilde{c}_t$ the sum of all transition counters, coincides on $[0,T]$ with the restriction of a counting path.

4. (Observation identity.) For all $\upsilon$ and $t\in[0,T]$:

5. (Control identity.) Let $K_t=\tilde{c}_t$ be the number of observation events up to time $t$, let $\tau_1<\dots<\tau_{K_T}$ be the jump times of the observation total in $[0,T]$, and for each $j\in\{1,\dots,K_T\}$ let $\upsilon_j\in\{1,\dots,\tilde{l}\}$ be the unique channel such that some observation counter with channel $\upsilon_j$ jumps at $\tau_j$; there is exactly one such channel, because by condition 3 the observation total jumps by exactly $1$ at $\tau_j$ while each observation counter is nondecreasing with integer values, so exactly one counter jumps at $\tau_j$. Then for every $t\in[0,T]$:

where the right-hand side is $h_0(t)$ on the event $K_t=0$.

6. (State identity.) For all $i$, all $\gamma$, and all $t\in[0,T]$:

Given a solution, the observation filtration $(\mathcal{G}_t)_{t\in[0,T]}$ is defined by letting $\mathcal{G}_t$ be the $\sigma$-algebra generated by the random variables $\Upsilon^\upsilon_s$ with $0\le s\le t$ and $\upsilon\in\{1,\dots,\tilde{l}\}$ together with every event of $\mathcal{F}$ of probability zero, and the system filtration $(\mathcal{F}^{\mathrm{sys}}_t)_{t\in[0,T]}$ is defined by letting $\mathcal{F}^{\mathrm{sys}}_t$ be the $\sigma$-algebra generated by the initial states $\varsigma^1_0,\dots,\varsigma^N_0$ and the random variables $N^{i,\sigma\gamma}_s$ and $\tilde{N}^{i,\upsilon}_s$ for $0\le s\le t$ and all indices, together with every event of $\mathcal{F}$ of probability zero. Each is a filtration with time index restricted to $[0,T]$, and $\mathcal{G}_t\subseteq\mathcal{F}^{\mathrm{sys}}_t$ for every $t$, since each $\Upsilon^\upsilon_s$ is a finite sum of the generating variables $\tilde{N}^{i,\upsilon}_s$ divided by $N$.