Michael Pratt
&
July 7, 2026
In the last few years there has been a massive resurgence in reinforcement learning as a post training step in the LLM training lifecycle, with two dominant applications of RL when it comes to LLM post training arising.
Reinforcement learning has been responsible for significant, tangible improvements in LLMs over the last few years and there is no doubt it still has a large role to play in the path to AGI.
In recent months there has been a shift in how LLMs are used, moving from chat-bots to agentic assistants that interact with humans, go away and perform tasks, before returning results back to humans, with this process repeating in a sequential stream of events. Interestingly, agents operating in sequential decision making processes align almost perfectly with the traditional theory of reinforcement learning. This provides us with an excellent opportunity to model these processes as MDPs and to use RL to accelerate our agents intelligence. At AgileRL, we have spent the last few months building out Arena’s capabilities to accommodate agentic reinforcement learning, and this article will shed some light on how we’ve approached such a complex system.
When an LLM operates as an agent, it sits in a loop with an external interactive environment: the environment accepts tokens generated by the LLM, and returns tokens back to the LLM; the LLM then produces new tokens, which are then accepted by the environment, and so on until a terminal state within the environment is reached. The sequential nature of this process naturally means that larger contexts are required and, as a result, memory usage is far greater than for the simpler RLVR case.
To accommodate these larger context lengths, we had to evaluate and optimize all of our algorithms to squeeze out every last drop of memory at each step. The table below includes some of the optimizations we have made to our algorithms for memory efficiency and suitability to long-context multi-turn RL:
GPU utilization has always been a challenge in reinforcement learning for LLMs. Rollout and training typically run sequentially, with each phase sitting idle while it waits for the other. For short-context, single-turn RL, this downtime is tolerable, but in the multi-turn case, where a single rollout involves multiple generation steps and expensive environment steps, it becomes a serious bottleneck. The obvious fix is to decouple the two phases: run rollout and training simultaneously, buffer the experiences collected by the rollout engine, and periodically sync weights to keep the two policies aligned. The catch is that the frontier algorithms for training LLMs are on-policy. The moment rollout and training run concurrently, the trainer is learning from experiences generated by slightly stale weights. Train on that mismatched data naively and the gradient estimates become biased and high variance, which in practice shows up as unstable learning, degraded sample efficiency, and in the worst case: outright policy collapse.
For training and rollout to run asynchronously, we needed an algorithm that could train on stale experiences without leading to policy collapse. CISPO, introduced by MiniMax towards the end of 2025, is built for exactly this. It keeps GRPO's group-based advantage estimation, which allows the value function to be dropped, but the loss is closer to REINFORCE than to a PPO-style surrogate. In the CISPO paper, the clipping mechanism in GRPO and PPO was found to adversely affect training. Tokens that initiate CoT reasoning (however, recheck, wait, aha) tend to have high ratios, because the base model assigns them low probabilities, and a small absolute probability change on a rare token is a large relative one. These tokens act as forks in reasoning paths and play a key role in stabilising entropy, yet they are the first casualties of the clipping interval.
CISPO's answer is to move the clipping. Rather than clipping the surrogate objective, it clips the importance sampling weight itself, and detaches it from the computation graph, so the per-token loss becomes a REINFORCE gradient scaled by a truncated, stop-gradient ratio. In PPO and GRPO, the min over the clipped and unclipped objectives means a token outside the trust region contributes exactly zero gradient. However, with CISPO, its weight is capped at the boundary instead. This means no token is ever discarded, only bounded.
This clipping makes CISPO a natural fit for asynchronous training. When rollouts are collected under a policy several optimiser steps behind the learner, ratio drift is the norm. Under PPO and GRPO, the tokens that get zeroed are precisely those where the current policy has moved furthest, and therefore the ones carrying the most information about the update direction. This means that the effective batch shrinks as staleness grows, and the update is only computed from whatever happened to remain inside the trust region. CISPO degrades differently: the gradient stays dense across the whole batch regardless of how far the behaviour policy has lagged. Both estimators are biased, since truncating importance weights is itself a bias, but a bounded contribution from every sample turns out to be far more benign than a full contribution from a censored one. The MiniMax authors also found the bound only really matters on one side. The damage from off-policy data comes from exploding weights, not vanishing ones, so ε_high can be set generously and the lower clip left effectively open.
The result of CISPO’s clipping is that the fork tokens survive. A "wait" or "recheck" token with a ratio of 3 still receives a positive update at the clip boundary, so reflective behaviour keeps being reinforced across the full window of stale experience. In our async setup, this is the property that holds entropy steady, even with training and rollout runs decoupled.
The architecture that consumes all of this is a pipeline of four components with a message bus in the middle, and the guiding principle is that nothing waits for anything else.
Rollouts are generated by a fleet of vLLM engines, each pinned to its own inference GPU. Every engine sits in a multi-turn loop with an agent environment running on CPU: the policy emits actions and tool calls, the environment executes them against its tasks and returns observations and rewards, and the loop repeats until the episode terminates. Because environments execute tools and arbitrary code, they run sandboxed, so failures in tool or code execution are contained rather than propagating to the rollout worker hosting them. Where stronger isolation is needed, environments can be dockerised and served from separate machines entirely, with the rollout loop talking to them through an env client.
Finished trajectories, tokens and rewards together, are published to the message bus. Putting a proper message bus between generation and training rather than a shared in-memory queue buys us durability and clean separation: rollout workers fire and forget, the trainer consumes at its own pace, and either side can restart or scale without the other noticing. Trajectories are streamed into a replay buffer on the trainer side, which holds only recent rollouts and evicts First-In-First-Out (FIFO). CISPO tolerates training on experience from a policy several versions old, but tolerance is not immunity, and the FIFO window puts a hard bound on how far behind the behaviour policy is allowed to fall. Old experience simply ages out before the trainer can see it.
The trainer samples batches from the buffer as grouped rollouts (complete groups of trajectories from the same prompt) since CISPO's group-based advantage estimation needs the full group to compute a baseline. Policy updates run on a separate pool of training GPUs, sized independently of the inference fleet. That independence is one of the quieter wins of the design: if generation is the bottleneck, inference GPUs can be added; if learning is the bottleneck, training GPUs can be added - neither change touches the other side of the broker.
The loop closes with weight synchronization. After each update, the trainer broadcasts the new LoRA adapter weights directly to the rollout engines over NCCL. Because we train adapters rather than the full model, the payload is megabytes instead of gigabytes, and the sync completes fast enough that the rollout engines never pause generation for it. The base model weights never move; only the adapter does. From the rollout engine's perspective the policy just quietly improves every few seconds while it keeps generating.
Taken together, the buffer decouples generation from training in time, the broker decouples them in space, and CISPO makes the resulting off-policy gap safe to learn from. Each piece covers a failure mode the others create, which is roughly what a system has to look like when the algorithm and the infrastructure are designed for each other rather than bolted together.
Everything described so far is a single agent: one policy, one trainer, one fleet of rollout engines. Arena treats that entire pipeline as a unit and runs several of them in parallel, each agent training with its own hyperparameter configuration. This is where AgileRL's evolutionary HPO comes in. At regular intervals the population is evaluated, the weakest agents are replaced with mutated copies of the strongest, and training continues without a restart. Hyperparameters that are notoriously sensitive in RL: learning rate, the CISPO clip bound, buffer window size, get tuned during the run rather than through a grid of sequential experiments, and compute that would have been spent on doomed configurations is reallocated to promising ones while training is still in progress. The result is that the architecture above scales in two directions at once: horizontally within an agent to generate and learn faster, and across the population to find the agent worth scaling.
We benchmarked Arena against CoreWeave's ART and Hugging Face's TRL on GEM's Sudoku (hard), training Gemma4-E4B-it with identical hyperparameters and set-up across every framework. We picked this environment deliberately: solving a hard Sudoku takes many turns of propose, check, and backtrack, and the accumulated board states and reasoning push context lengths towards the 32k limit, so it stresses exactly the long-horizon, long-context regime this article has been about. Arena ran in both configurations described above: fully asynchronous, and a synchronous mode where rollout and training alternate on the same GPUs.
Neither baseline learned the task. ART's score stayed flat near zero for the full run, and TRL degenerated after roughly 50 steps and never recovered. The completion length curves show what actually went wrong: both frameworks' average response lengths collapsed towards zero, which is the entropy failure mode from earlier in this article playing out in practice. A policy that stops generating the tokens that open up longer reasoning paths cannot work through a board that takes dozens of steps to solve, so the completion collapse and the score collapse are the same event. Episodes get shorter, rewards get worse, and the run finishes early with nothing to show for it. Both Arena configurations went the other way, growing their completions to around 30k tokens, comfortably inside the 32k limit, while their scores climbed. Async finished at 0.85 and sync at 0.77.
The per-step and wall-clock views tell slightly different stories. Per step, sync learns fastest early on, since every batch is fully on-policy. Async starts slower while the buffer fills with experience, then overtakes. On wall clock, which is the axis anyone paying for GPUs actually cares about, async reached its final score in about 16 hours against sync's 24, because it never spends time with either side of the pipeline idle. Training throughput makes the mechanism visible: async climbs to nearly 800 tokens/s as completions lengthen, sync holds around 300, TRL managed about 150 before degenerating, and ART sat near 50 throughout.
The cost numbers compound all of this. The Arena runs completed on a single A100 40GB, $118 for async and $88 for sync, with the QLoRA and offloading work from the optimization table doing the heavy lifting on memory. ART and TRL both needed an 80GB GPU, and at their measured throughputs would take roughly 160 and 48 hours respectively to process the same number of tokens as the Arena run, costing $811 and $243. That comparison is generous to the baselines, since it prices the compute as if their runs succeeded, and neither did. Cheaper hardware, a fraction of the time, and the only frameworks in the comparison that actually learned the task.
Every Arena result above comes from a single agent with hyperparameters we already knew worked on this environment. That's not the position most teams are in. On a new task the learning rate, clip bound, and buffer window that produce the curves above are unknown, and in RL the penalty for guessing wrong isn't a slightly worse score, it's the collapse ART and TRL demonstrated. The usual answer is a sweep: run several configurations, keep the best, absorb the calendar time.
Arena's evolutionary HPO runs that search as a population instead. We repeated the benchmark with a population of four async agents starting from shared hyperparameters, evaluated at intervals, with the weakest replaced by mutated copies of the strongest. The population climbed nearly as steeply as sync from the first steps, since evolution reallocates compute away from configurations that learn poorly under staleness, and passed 0.65 within 5 hours, ground the single async run took 16 hours to cover, finishing level with it around 0.8. Per agent, throughput was identical to the standalone run; the layer adds no overhead.
Four agents cost four times a single agent, $472 for this run. The alternative of running a hyperparameter sweep would come to the same $472 for the same four configurations, spread over roughly 64 hours of wall clock, where the population reached a strong policy in 5. Even accounting for the multiplier, the entire search cost less than the $811 ART spent on a single run that collapsed. Plus, on a new environment, where an ill-chosen learning rate or clip bound invites exactly the collapse the baselines demonstrated, the population compresses a week of sequential search into the length of a single run.
The thread running through all of this is that the algorithm and the infrastructure are not separable decisions. CISPO only pays off if the system can actually deliver stale experience at high throughput, and the async architecture is only safe because CISPO makes the resulting off-policy gap learnable. Build either half in isolation and you get one of the failure modes in the benchmark: a stable trainer starved of tokens, or a fast pipeline feeding a policy that collapses. The Sudoku results are what happens when the two halves are designed together: an impressive final score on a task the baselines couldn't learn at all, on cheaper hardware, in 16 hours.
There is plenty left to do. We are continuing to build for longer horizons than Sudoku, environments where tool latency dominates generation, and optimizing population-based evolutionary training on frontier-scale tasks. But the shape of the system is now settled: treat the agent loop as an MDP, keep every GPU busy, and use an objective that treats off-policy data as the normal case rather than a nuisance. If you're training agents and fighting the same bottlenecks, Arena is how we'd suggest fighting them.
If you are building multi-turn agents and want to talk through what the right setup looks like for your problem, get in touch.
Check out Arena, and get started training agents for free.
