Nick Ustaran-Anderegg
&
November 9, 2026
In October we benchmarked Arena against Prime-RL, Miles, TRL, verl and ART on Agent World Model, training Qwen3.6-35B-A3B with CISPO on one 8x H100-80 node. Arena already outperformed other frameworks in accuracy and speed. It finished that job in 35.4 hours and scored 40.1% on 668 tasks from environments the agent never saw during training.
We ran the same job on Arena again, after bringing in some significant optimizations that we have been working on for a while. It finished in just 11.6 hours and scored 42.2%, outperforming the previous Arena run, all other frameworks, and Claude Opus 5.5. Median step time went from about 22 minutes to about 7.4 minutes. Generated tokens per second went from 1,584 to 2,667.
The model, the environment, the 96 steps, and the RL hyperparameters were identical to the October post. However, Arena now resolves the training and generation setup at the start of every run: parallelism, how many GPUs train and how many generate, micro-batch size as a token budget, checkpointing, each rollout engine's tensor-parallel width, and other settings that only affect speed and memory. We have also introduced new optimized kernels and fused operations that provide further speed improvements.
The orange line is the latest Arena run. The faded dashed line is October Arena. Prime-RL, Miles, TRL, verl and ART are the other frameworks benchmarked. Training reward on the new run sits above the October Arena curve for most of the 96 steps. The new run is done before the October Arena job has finished a third of its steps.
Against the other libraries the gap is larger. Prime-RL was the only one that approached Arena's reward. Miles, TRL, verl and ART stay well below. Prime-RL, Miles and ART were stopped after 48-hours because they still had not finished. Arena reaches a higher reward than any of them in 11.6 hours, then stops because the job is over.
Arena is now 42.2% on that harness, 0.8 points above Claude Opus 5.5 (41.4%) and 8.3 points above Prime-RL, the next library. The October Arena checkpoint was 40.1%. Miles is 23.6%, TRL 21.8%, Claude Sonnet 5.5 19.1%, ART 17.5%, verl 16.6%, and the base model 10.0%. After this run, a 35-billion-parameter model with 3B active parameters is ahead of Opus 5.5, and 23.1 points above Sonnet 5.5, on tasks from environments it never saw in training.
An asynchronous RL job has a long list of settings that do not change the objective: the parallelism techniques to use (data, tensor, expert, context) and their values, how large a packed micro-batch to run, gradient checkpointing, how to split eight GPUs between the trainer and vLLM, each generation engine's configuration, the engine scheduler step, and more. Pinning them by hand means knowing the model, the GPU, and the memory math well enough that a wrong value either OOMs or leaves the node idle.
Arena searches those at the start of the run and automatically selects the optimal setup. The planner knows the model, task and the GPU type; it takes the fastest plan that fits, and it gives up speed only when the faster setting does not: gathered FSDP parameters before resharding, a large expert chunk before a small one, offload last.
The planner also splits the GPUs between rollout and training. It uses fitted cost models for learn time and generation time, based on analytical solutions and benchmark runs. On Qwen3.6-35B-A3B CISPO it picks four trainers and two TP2 engines at rollout batch 16. Coincidentally, all other previous runs also used the same split (4 GPUs for training, 4 GPUs for generation).
Automatically using the best training configuration means that you get the most out of your compute. GPU hours are not wasted on non-optimal setups, saving time and money; snooze, you lose.
Async-RL splits work between the parallel generation of tokens to learn from, and the actual weight updates based on those tokens. Both sides need to be fast individually, and balanced in their generation and consumption, in order to train as quickly as possible.
A large proportion of the learn-side of the async system is often taken up by work that cannot change the policy. When completions offer no useful training signal, GPUs are left idling for one another, forcing you to incur extra wall-clock time even though the loss remains unchanged. Arena eliminates this waste: compute is devoted strictly to tokens involved in the update, GPUs remain synchronized, and gradient-free groups are bypassed completely. The objective remains unchanged, but each step is significantly more efficient. Consequently, the median step time for this job dropped from approximately 22 minutes to around 7.4 minutes.
The other side of the async system, generation, also experiences inefficiencies during a run. Rollout engines often generate excess samples beyond what the trainer requires or stall completely during policy updates, incurring costs for tokens that ultimately contribute nothing to model training. Arena addresses this by tightly coupling generation with the trainer: it prevents the execution of rollouts that would be rendered redundant by weight updates, and eliminates engine pauses during parameter shifts. As a result, a significantly higher proportion of wall-clock time is spent producing gradient-generating tokens.
Arena's performance stays above every library on this chart for the whole 11.6 hours. The model, the environment, and the 96-step budget are the same for every stack in this comparison, and what differs is how the job is run. Arena finishes in 11.6 hours at 42.2% held-out, above Opus 5.5. The other libraries take much longer and land well below that.
Agent World Model is fully integrated into Arena. All recent updates, including the planner, learn-step kernel enhancements, and generation optimizations, are deployed directly on the platform, allowing new experiments to leverage them automatically without requiring pinned memory configurations. Details regarding the environment, library comparisons, and evaluation protocols can be found in our October post.
Ready to build? To recreate this run or train a custom MCP tool-calling agent, simply create an experiment, pick Agent World Model along with your target model, and start training. For additional guidance, visit our Docs or Book a demo.