Comparing Online Agents on Shared Spread Paths

A policy playback explains decisions on one path. A learning comparison needs repeated runs, a common budget and held-out evaluation. These two recordings show those views of the same synthetic OU experiment.

A compatible comparison cohort

The study compares DQN, C51, QR-DQN, maximum-entropy Soft-Q and categorical SAC. Each can consume the same online one-step transition stream.

Each algorithm has five independently initialized training runs. Every run receives 64 training episodes of 32 steps. Following a 128-transition warm-up, one update per new transition gives 1,921 updates. Evaluation uses the same 16 held-out path seeds at every checkpoint without changing the model.

The right panel adds Gaussian sensor noise with standard deviation 0.15 at inference time. Shaded regions show normal 95% intervals across the five training seeds.

PPO, APPO and IMPALA need behavior-policy trajectories; BC, MARWIL and CQL need a declared offline dataset; DreamerV3 needs a sequence and model-compute budget. They do not belong on this particular transition-budget curve without a separate comparison design.

Recorded final results

The repository records the following results on 2 August 2026. Values are synthetic episode rewards under the experiment’s stated objective.

The deterministic threshold reference returns 2.8367 on these paths. Soft-Q has the highest clean point estimate and SAC the highest noisy point estimate, but several intervals overlap. Some noisy point estimates improve in this threshold-like environment. These observations do not establish a general ranking or general robustness to noise.

Decisions on one held-out path

The playback selects, for each algorithm, the training run closest to that algorithm’s median final clean return. Every selected policy then sees the same exogenous path. The action raster shows SHORT, FLAT and LONG; the lower panel accumulates realized reward.

This path illustrates behavior. The repeated-run learning curves and recorded intervals supply the comparison evidence.

Reproduce the study

PYTHONPATH=src python examples/compare_ou_algorithms.py \
  --runs 5 --train-episodes 64 --eval-interval 4 \
  --eval-episodes 16 --horizon 32 \
  --json-output docs/latex/figures/ou_algorithm_convergence.json \
  --plot-output docs/latex/figures/ou_algorithm_convergence.pdf \
  --convergence-gif-output docs/markup/algorithm-convergence.gif \
  --playback-gif-output docs/markup/algorithm-playback.gif

Source and related material

• Protocol, final results and limitations.
• Comparison implementation.
• Recorded aggregate data.
• Distributed spread-learning walkthrough.