Distributed Learning on a Synthetic Spread

The recording exposes a distributed learning loop on a public synthetic process. Two actors send experiences to a central C51 learner. The terminal tracks replay stores, policy versions, training losses and held-out results while the run progresses.

Process, observation and action

The spread follows the discrete mean-reverting process

\[x_{t+1}=x_t-\kappa x_t+\sigma\epsilon_t,\qquad \epsilon_t\sim\mathcal N(0,1),\]

with default \(\kappa=0.18\) and \(\sigma=0.35\). Observations contain the spread z-score, its last change in standardized units, the current position and the remaining episode fraction.

The action vocabulary is SHORT, FLAT and LONG. An action selects the position held over the next spread change. The environment is synthetic; the spread does not represent an executable pair of securities.

Reward and the fixed reference

Let \(p_t\) be the chosen position and \(s\) the stationary spread standard deviation. The reward is

\[r_t=p_t\frac{x_{t+1}-x_t}{s} -0.035|p_t-p_{t-1}|-0.004|p_t|.\]

Turnover and position penalties are explicit parts of the learning objective. The same-path non-learning reference is short above \(+0.65z\), long below \(-0.65z\) and flat otherwise.

The screen reports cumulative reward, reference reward and their difference, maximum drawdown of cumulative reward, turnover and the fraction of positive episodes. These are metrics in the declared synthetic environment, rather than currency returns.

Training and held-out experience

The fixed episode schedule interleaves four training episodes with one validation episode. Split assignment is fixed before an episode starts and carried with every transition.

Training samples enter prioritized replay and update the training-only normalizer. Validation samples use a separate store and do not update either the optimizer or normalization state. Training includes epsilon-greedy exploration; validation uses deterministic actions.

The auxiliary head predicts the sign of the process’s conditional drift. Labels arrive after transition batches, exercising the delayed-label join. The terminal’s policy probe at five spread z-scores makes the currently selected actions visible.

Run and retain the artifacts

python examples/train_ou_spread.py

For a bounded machine-readable run:

python examples/train_ou_spread.py \
  --json --train-episodes 8 --validation-episodes 2 --horizon 24

Retain the run manifest, episode results, summary and checkpoint with:

python examples/train_ou_spread.py \
  --output-dir /tmp/jormungandr-ou-spread

Fixed path seeds and split assignments make the environment reproducible. Concurrent actor completion can still affect learner updates and the fitted policy; a fixed synthetic seed should not be presented as a guarantee of identical learned parameters.

Source and related material

• OU process, reward and service boundary.
• Trainer source.
• Recorded functional results.
• Comparing online agents on shared paths.