Three Upgrades to AgileRL's Evolutionary HPO

Reinforcement learning is very sensitive to hyperparameters and network architecture. Moreover, an RL agent generates its own training data, so a hyperparameter choice changes what the agent learns from, making hyperparameter optimisation (HPO) in RL extremely complex. A learning rate that works at the start of training is often not optimal a few steps later, and the right network size for an early policy is rarely the right size for a converged one.

This is why AgileRL builds hyperparameter optimisation into training rather than around it. A population of agents trains in parallel, each with its own hyperparameters and architecture. Every evo_steps, they are ranked on performance, the strongest configurations are carried forward into the weaker slots, and hyperparameter, network architecture, parameter, and activation mutations allow for a thorough exploration of the configuration search space. The output is not a single setting you commit to for the whole run, but a schedule that adapts to the different training stages.

We have just added three upgrades to that loop:

1. Multiple-Frequencies Population-Based Training (MF-PBT), a second selection strategy that consctructs multiple subpopulations which evolve at different frequencies.
2. ReGraMa, a targeted parameter mutation that resets neurons that have stopped learning.
3. Function-preserving architecture mutations, which grow the neural networks without disturbing the learned policy.

‍

Multiple-Frequencies Population-Based Training

During HPO, evolution happens every evo_steps, which is a fixed value, and that number forces a trade-off:

• Set it low and you evolve often. The hyperparameter schedule becomes fine-grained, but each mutated agent is trained for a short period of time, which does not allow long-term benefitial mutations to stand out. Any perturbation with a delayed payoff gets ranked below those with an immediate one, and is replaced before it can demonstrate otherwise. As a result, short-term gains dominate and the best fitness ends up converging to a local minimum.
• Set evo_steps high and the rankings become more reliable, because every agent has had time to learn and show the performance of its configuration, but the schedule is coarse, the population adapts slowly, and you spend compute training configurations that stopped being the right ones a while ago.

There is no single evo_steps value that gets you both. Multiple-Frequencies Population-Based Training (MF-PBT) addresses this fundamental problem.

In MF-PBT, the population is split into subpopulations, and each one evolves at a different multiple of the base cycle, which happens every evo_steps. For example, with two subpopulations at ratios [1, 5], the first evolves every cycle and the second evolves every fifth. The granularity of the overall schedule is the same (as tournament selection with evo_steps) thanks to the first subpopulation. What changes is that the fast subpopulation adapts on a short horizon while the slow one trains promising configurations for enough time to find out whether they pay off in the long term.

Within each subpopulation, top performers are cloned over the weakest slots and mutated. Between subpopulations, strong agents from other subpopulations migrate into slots that are underperforming, so a good result found on one frequency spreads to the other. Migration is deliberately asymmetric: an agent arriving from a faster-evolving subpopulation brings its networks but adopts the destination subpopulation elite's hyperparameters. Good weights spread freely, while the fast, aggressive schedule that produced them does not get imported into the subpopulation whose job is to be patient. On the other side, a full clone is migrated from a slower subpopulation into a faster one.
‍

‍

How to Use MF-PBT

Both tournament selection and multi-frequency selection (i.e., MF-PBT) are supported in AgileRL's framework. The manifest field tournament_selection has been renamed to selection_strategy, and takes either strategy: tournament or strategy: multi_frequency:

training:
  pop_size: 16

selection_strategy:
  strategy: multi_frequency
  n_subpopulations: 2
  evolution_frequency_ratios: [1, 5]
  n_winners: 2
  n_survivors: 0
  n_open_for_migration: 2
  n_losers: 4

‍

The new parameters for MF-PBT are explained below:

‍

Note that, to use MF-PBT, pop_size must be at least 6, and must divide evenly by n_subpopulations. See the MF-PBT tutorial for a full walkthrough.

MF-PBT performs best with a large population size (pop_size ≥ 16), since two subpopulations of 8 agents each is the minimum to keep exploring the hyperparameter and network architecture space effectively while performing migrations and allocating more compute to promising configurations. For smaller populations, tournament selection remains a solid alternative. Both are one line apart in the manifest, so either is easy to try on your own task.

‍

ReGraMa

Parameter mutations directly perturb the network weights to explore new policies during training. This is done by Gaussian noise perturbations and resets performed on randomly selected weights. On top of that, ReGraMa is now used during parameter mutations too: ReGraMa is a targeted kind of parameter mutations.

The target is dormant neurons. Over the course of training in reinforcement learning, some units' gradients fade towards zero and they stop learning. Nothing visibly breaks, the agent keeps training, but those dormant units still consume compute, so the network you are paying for ends up larger than the network that is actually learning.

ReGraMa identifies those units and puts them back to work. A revived dormant neuron gets fresh incoming weights, so it can learn something new, and its outgoing weights are initialised to non-zero values small enough so that the agent's behaviour is barely disturbed whilst the reset neuron starts learning fast. In other words, it starts receiving gradients immediately and contributes again, rather than being flagged dormant a second time on the next pass.

How to Use ReGraMa

ReGraMa resets are automatically performed as part of parameter mutations in AgileRL's framework. However, their behaviour can be configured by the user via the dormant_threshold parameter in the manifests.

A neuron's dormancy score is measured relative to its own layer, so dormant_threshold is a relative quantity. At the default of 0.01, a unit counts as dormant when it is receiving roughly 1% of the gradient its average neighbour receives.  We would suggest leaving it at 0.01. If you do want to tune it, its impact is straightforward: raising it revives more neurons per mutation, resetting more capacity at the cost of perturbing the policy more, while lowering it revives only the units that have stopped learning most noticeably.

‍

Function-Preserving Architecture Mutations

Architecture mutations modify the size of the networks mid-training, so agents are not stuck with whatever capacity they had at the beginning. These mutations allow architectures to be optimised during training alongside hyperparameters.

However, adding or removing neurons or a whole layer to a trained network is usually destructive, as the policy changes abruptly. Therefore, the fitness value drops and the mutated agent is not able to survive the next tournament selection round, not giving new capacity the chance to learn anything useful.

Thus, function-preserving addition operations have been implemented in AgileRL's framework. When possible, capacity is added, but the policy's behaviour at the moment of mutation, does not change. This is achieved as follows:

• When adding a node, the new unit gets randomly initialised incoming weights, but its outgoing weights are faded to near-zero values. The layer's output is unchanged, so the agent performs exactly as it did before, and the new unit only begins influencing the policy as training gradually guides it.
• New layers are initialised as an identity matrix (Net2DeeperNet operation), passing its input straight through. Performance is preserved at the moment of insertion, and gradients move the layer away from the identity during training.

Note that removal operations keep their existing behaviour, since removing capacity can't guarantee that the function the network represents is preserved.

How to Use Function-Preserving Architecture Mutations

Function-preserving node and layer addition operations are carried out automatically. There's no new manifest field and nothing to switch on.

Preservation applies wherever the architecture supports it and architecture mutations fall back to the original random initialisation strategy everywhere else. Function preservation while adding capacity is guaranteed when there are no normalisation layers between a widened layer and its activation, no cross-unit activation functions are used (i.e., Softmax), and the affected layer is not part of a multi-input encoder, residual block or RNN. Moreover, function-preserving layer additions also need the involved activation to be ReLU or Identity, and the new layer to be square.

‍

Impact of the Upgrades

The plot below represents the evaluation score (i.e., mean of the best fitness in the population) over the number of per-agent environment steps, before and after the upgrades were implemented:

‍

The value of the evaluation score that matters here is the last one, after training has concluded. HPO produces a population, but you deploy the best agent in it at the end of the run. According to this benchmark, the upgrades increase the average performance of the deployed agents by 6%. Note that both lines outperform an expert policy within 4.5 million steps, showing the effectiveness of the HPO strategy implemented in AgileRL's framework.

As seen above, the upgrades lead to a consistent improvement over the baseline. This margin results from a cumulative effect related to:

• MF-PBT allocating compute to promising configurations which pay off in the long term while returning a fine-grained schedule. In contrast, in the baseline, PBT prioritises short-term improvements.
• ReGraMa keeping the amount of dormant neurons low during training, leveraging the full potential of the neural networks that are being used.
• Function-preserving layer and node additions facilitating the survival of the networks whose size has just been modified. This extra capacity, when trained, helps widen the margin between both lines.

The three upgrades are already available in AgileRL's open-source framework. Update now and try them.