Stochastic Integral Reinforcement Learning for Distributed $H_\infty$ Optimal Formation Tracking of Multi-Agent Wheeled Mobile Robots

Khanh Thai¹, Huy Dao¹, Nga Thi-Thuy Vu^1*

¹School of Electrical & Electronic Engineering, Hanoi University of Science and Technology (HUST), Vietnam

^*Corresponding author

Paper Code Coming soon

Stochastic $H_\infty$ zero-sum game Critic-only online IRL Model-free learning Mean-square UUB guarantee

Top: the virtual leader's serpentine reference path (dashed) and the three agents' trajectories over 50 seconds, at equal x/y scale, with the communication graph drawn at the 5, 15, 25, 35 and 50 second snapshots and the locked formation at the end. Bottom: the same five snapshots drawn leader-relative and magnified, each against the dashed target shape - flat and far from the target at 5 seconds, sitting exactly on it from 25 seconds onward. — The full 50-s experiment at a glance: three wheeled mobile robots learn, online, to track a serpentine virtual leader over a fixed communication graph — converging to a rigid formation under Itô process noise and worst-case disturbances.

Overview

A genuinely stochastic $H_\infty$ formation-tracking game, solved online.

We cast distributed formation tracking for multi-agent wheeled mobile robots as a stochastic zero-sum $H_\infty$ game — each agent driven by a genuine Itô SDE rather than an ODE — and solve the resulting coupled stochastic HJI equations online with a critic-only, model-free reinforcement learning scheme. Simulation confirms the guaranteed mean-square-bounded tracking in real time.

Genuinely Stochastic $H_\infty$ Game

Agents follow Itô SDEs, not ODEs — the coupled HJI equations gain a genuine Itô-correction term that vanishes exactly as the noise does.

Critic-Only Online IRL

One critic per agent yields both the policy and the worst-case disturbance in closed form; a historical-data adaptive law replaces persistence of excitation with a checkable rank test.

Model-Free Learning

An off-policy reformulation lumps the unknown input and diffusion channels into the agents' own policies — a change of regressor, not of algorithm.

Mean-Square UUB Guarantee

Tracking and critic-weight errors are provably bounded in the mean-square sense, recovering the deterministic result exactly as $\Sigma_i \to 0$.

MuJoCo visualization of the paper's experiment: the virtual leader (gold beacon) traces the serpentine reference while three follower WMRs self-assemble into the learned rigid formation. White links are the live communication-graph edges.

Method

Coupled stochastic HJI equations, solved with critic-only IRL.

We cast distributed formation tracking as a stochastic zero-sum game and solve the resulting coupled stochastic HJI equations online with a critic-only integral reinforcement learning (IRL) scheme. Historical data replace persistence-of-excitation requirements, and the controller does not need the drift dynamics of the augmented multi-agent system.

SIRL overview in three panels: (a) the per-agent online critic-only IRL / H-infinity control loop, (b) the critic regressor and history-stack construction, and (c) the model-free off-policy extension — **SIRL overview.** (a) Per-agent loop: the neighbourhood error drives a critic whose Bellman residual and replayed history feed the adaptive law (eq. 35), producing the $H_\infty$ policy and the worst-case disturbance. (b) The critic bases $\phi_i(\delta_i)$ are recorded into a fixed-size history stack $Z_i$, whose rank condition replaces persistence of excitation. (c) The model-free extension lumps the unknown $g_i,k_i$ into an enlarged parameter $\mathcal{Z}_i$; the drift $f_{ei}$ and diffusion $\Sigma_i$ cancel against the Bellman identity and are never identified. *Click to open full size.*

Theorem 1 (Mean-Square UUB)

Under the adaptive law (eq. 35) and the history-stack rank condition — with no further hypothesis on the closed loop — the tracking error $\delta_i$ and critic-weight error $\tilde W_i$ are ultimately uniformly bounded in the mean-square sense, with a bound that reduces exactly to the deterministic pathwise result as $\Sigma_i \to 0$. The learned value, policy and worst-case disturbance approach the Nash equilibrium to within a residual set — $\mathbb{E}\|V_i^* - \hat V_i\|^2 < \epsilon_{V_i}$, $\mathbb{E}\|u_i^* - \hat u_i\|^2 < \epsilon_{u_i}$, $\mathbb{E}\|\vartheta_i^* - \hat\vartheta_i\|^2 < \epsilon_{\vartheta_i}$ — which shrinks as the critic basis is refined, not as the noise vanishes.

Communication graph — Communication topology

Results

Accurate formation tracking under noise and adversarial disturbances.

The proposed SIRL controller achieves accurate formation tracking for wheeled mobile robots under process noise and adversarial disturbances. Position and orientation errors converge, while critic network weights remain bounded during online learning.

Position error ex — $e_x$ tracking error

Position error ey — $e_y$ tracking error

Orientation error — $e_\theta$ tracking error

x trajectories of leader and followers — $x_i\ (i=1,2,3)$ and $x_0$

y trajectories of leader and followers — $y_i\ (i=1,2,3)$ and $y_0$

Heading trajectories of leader and followers — $\theta_i\ (i=1,2,3)$ and $\theta_0$

Linear velocity control input — Control input $v_i$

Angular velocity control input — Control input $\omega_i$

Estimated worst-case disturbance policy $\hat\vartheta_i$, dynamic-channel component. Left: full horizon, dominated by the exploration signal over $[0,5]\,$s. Right: the settled post-excitation behaviour.

Online Critic Learning

Critic weights converge online — no persistence of excitation required.

Critic neural-network weights converge online using historical data, without requiring persistence of excitation. Ablation studies quantify the contribution of the reinforcement learning component.

Critic weights W1 — Critic weights $W_1$

Critic weights W2 — Critic weights $W_2$

Critic weights W3 — Critic weights $W_3$

Ablation study — Ablation: contribution of the RL component

Disabling each component in turn, with topology, reference, noise realisation, and seed held fixed (mean position-error norm $\bar{\|\delta\|}$ over $t>25\,$s; cost $J=\sum_i\int_5^{50}(e_i^TQ_ie_i+u_i^{rT}R_{ii}u_i^r)\,dt$):

Variant	Cost $J$	$\bar{\\|\delta\\|}$ (m)
Proposed (full)	4.38	0.026
Without Bellman-residual / replay terms	4.49	0.025
Without stabilizing term $\xi_i\Theta_i$	diverges	—
Critic weights frozen at $\hat W_i(0)$	diverges	—
Without learned policy $u_i^r$	$1.5\times10^{8}$	14.11

The model-based feedforward alone cannot stabilise the formation (error grows past 14 m without the learned policy); the stabilizing term is what keeps the closed loop out of the divergent region; and the Bellman-residual / experience-replay terms lower the cost by a further $\sim$2.4% at equal tracking accuracy.

Full critic-only IRL (left) versus the fixed feedforward alone, learned policy $u_i^r$ switched off (right). This lightweight demo plant carries safety caps that prevent outright divergence, so what is visible is the milder form of the same effect: a looser, laggier formation without the learned policy.

Model-Free Extension

The same algorithm, run with no model at all.

Reading the stationarity conditions backwards lumps the unknown products $\nabla V_i^Tg_i$ and $\nabla V_i^Tk_i$ into the agent's own policies; the graphical-game cross terms $\nabla V_i^Tg_j$ are identified from data. The learning mechanism — residual, history stack, adaptive law — is untouched.

Nothing About the Plant Is Known

Not just the drift $f_{ei}$ — the input channels $g_i,k_i$ and the diffusion $\Sigma_i$ are unknown too, and no diffusion parameter is ever identified.

Exact, Not Approximate

The lumping matches the model-based regressor to $\sim\!10^{-15}$ window by window — exact, not an approximation licensed by small residuals.

Cheaper, Despite More Parameters

$0.89\,$ms per step against $1.72$–$1.81\,$ms model-based, while carrying up to $255$ parameters versus $15$: the extra parameters cost less than the matrix triples they replace.

Same Guarantee, Still Learning

Theorem 1 carries over unchanged (Corollary 1), and online adaptation still cuts closed-loop cost by $19\%$ against a frozen initialization — with no model at all.

Normalised singular-value spectra of the model-free regressor for the three agents, comparing independent exploration, shared exploration, and the model-based regressor — **The rank condition is attainable — but demands independent excitation.** Singular-value spectra of $\Lambda_i$ ($\dim(\mathcal{Z}_i)=95,255,175$) against the 15-dimensional model-based $\Delta\phi_i$. Solid: independent exploration reaches full rank exactly. Dashed: probing the disturbance channel with a scaled copy of the control probe — the natural shortcut — fails the rank condition no matter how long the run.

Online closed-loop run of the model-free algorithm: formation tracking error, model-free parameter norm, and learned actor output — **Online closed-loop run with $f_{ei},g_i,k_i,\Sigma_i$ all unknown.** Left: tracking error, settling to the centimetre scale after exploration is withdrawn at $25\,$s. Centre: $\|\mathcal{Z}_i\|$, still rising for agent 2 (coupled to both neighbours) — the signature of ultimate boundedness under a finite basis, not convergence. Right: the learned actor output $\hat u_i$.

The model-free run of Algorithm 2, with $f_{ei},g_i,k_i,\Sigma_i$ all unavailable to the learner. The unruly opening is the independent exploration the rank condition demands, not instability; probes are withdrawn at $25\,$s, after which the error settles to $0.014$, $0.009$ and $0.006\,$m at $50\,$s.

One caveat worth stating: the actor bases must depend on the measured configuration, not the neighbourhood error alone. For a nonholonomic vehicle the ideal actor carries the heading through $S(\theta_i)$, so the default basis in $\delta_i$ alone leaves relative errors of $24.1\%$, $29.3\%$ and $17.3\%$.

Citation

Cite SIRL

@misc{thai2026sirl,
  title={Stochastic Integral Reinforcement Learning for Distributed $H_\infty$ Optimal Formation Tracking of Multi-Agent Wheeled Mobile Robots},
  author={Thai, Khanh and Dao, Huy and Vu, Nga Thi-Thuy},
  year={2026}
}