Michael Pratt
&
October 20, 2026
In the last few months at AgileRL we have been working hard to expand the capabilities of our agent training platform, Arena, to bring our customers cutting edge reinforcement fine-tuning for the most capable open-weight models. In this blog post we present a new set of Arena benchmarks trained using Agent World Model, an open source tool calling environment developed by Snowflake and UC San Diego that scales to 1,000 unique environments spanning real world scenarios such as shopping, social media, finance, and travel. Each environment is a fully functional Model Context Protocol (MCP) server with tools, database state, and verification logic and has been proven to provide excellent out-of-distribution performance on other tool calling tasks to models trained on it.
We trained Qwen/Qwen3.6-35B-A3B on this environment using a variety of the top open-source RL libraries to prove that training with Arena yields significantly better results than any of the open-source alternatives. We chose this model specifically as it shows strong agentic benchmarks, is cheap to serve given its MoE architecture with only 3B active parameters, and its native support of long context lengths which make it ideal for long-context multi-turn trajectories typical of tool calling rollouts.
Our results show that training an agentic tool-calling model in Arena yields the best results when compared with SOTA open-source, as well as beating Claude Sonnet 5.5 and achieving 97% of the performance of Opus 5.5.
The Agent World Model environment is now live in Arena. To replicate our results or train your own MCP tool-calling agent, create an experiment, select Agent World Model as the environment, choose a model and click Train.
We benchmarked Arena against five open-source RL libraries: Prime-RL, Miles, TRL, verl and ART. For a fair comparison across libraries, we selected each library’s SOTA group-based RL algorithm, trained using Parameter Efficient Fine-Tuning (LoRA) and used the hyperparameters that were selected by the authors in the AWM paper. Every experiment was trained on 8 x H100 nodes using async rollout with a 50:50 rollout/training GPU split. For all experiments we used a single rollout engine distributed across 4 GPUs using tensor parallelism and the training network was distributed using expert-parallelism.
We trained on the same subset of data used in the paper: 3,315 tasks from 526 environments, roughly a third of the full set. We then randomly sampled 10% of the remaining 6,685 tasks which are comprised of environments the agent never saw during training to collect our evals.
For each RL library we trained on the same subset of tasks, with each task running for a maximum of 20 turns. The agent receives the task in the form of a user message, then reasons and makes one tool call per turn, reading JSON output from the environment to decide its next step. Rollout ends when the max number of turns is reached or an agent responds without a tool call.
The Agent World Model simulation consists of a number of environments, each of which is a simplified app or website, such as an online shop, CRM bank, smart home controller, etc. implemented as a python MCP server exposing a number of read, create, update, and delete tools and backed by its own SQLite database of roughly 18 tables. The agent never sees the tools directly, instead it gets two meta-tools: list_tools, which it must call exactly once at the start to retrieve every tool’s description and schemas, and call_tool, which runs a named tool with JSON arguments. This lets a single agent operate across all environments without modification.
Every task in this environment ships with its own python verification function that compares the database state before the task has begun vs after and gives a score of +1 on successful completion. If the verification function fails, an invalid tool is called, or a valid tool is called with incorrect arguments the agent receives a score of -1. Episodes that fail because of an environment problem, such as a server error or timeout, receive 0, so the agent is not penalized for faults outside its control.
Below are a few examples of some of the tasks in Agent World Model:
While other libraries demonstrate some level of learning on the task, Arena’s speed and robustness allow the agent to quickly grasp the task at hand, leading to the largest and most stable training reward, for the fewest GPU hours in the 96 training steps we trained for. We decided to employ a training cut-off period of 48 hours to avoid cloud costs spiraling out-of-control and, of the libraries we benchmarked against, Prime-RL, Miles, and ART all had to be cut short.
Prime-RL was the only library out of the open-source offerings that came close to the reward achieved by Arena however it was at the cost of training time and significant reward variance, taking an additional 96 GPU hours to run 33 fewer training steps. If we hadn’t cut the Prime-RL run short, to complete the same number of steps that were completed in Arena it would’ve taken an additional 25 hours of training time equating to roughly $3,300 of additional compute costs.
One of the main issues we came across with the open-source libraries was their sample inefficiency. When learning with group-based algorithms, groups with no reward variance lead to zero advantage and the sample contributes zero to the policy loss, nullifying that term from the loss function entirely. In cases where KL regularization is not used, the entire loss term becomes zero and there is no gradient to update the network. To combat this many of the open source libraries dynamically filter groups to learn only on samples that have reward variance across a group. We noticed that this mechanism led to much longer learn-steps, especially later in training as the policy improved and rewards were more likely to be the same, leading to many rollout samples being dropped. Time is then wasted generating extra groups to replace the ones that have been dropped by the dynamic sampling mechanism.
So why don’t we hit this problem in Arena? Because we stay at the bleeding edge, regularly reviewing the papers for new optimizations to keep our training stack at the frontier. We have implemented two key mechanisms to mitigate performance degradation while still dynamically filtering zero advantage groups. The first method involves keeping a per-task store of trajectories with varying rewards. When the replay buffer receives a group of trajectories where every member has the same reward, a single different reward trajectory for that same task can be switched into the group. Advantage is no longer zero and the group will yield a gradient that we can learn from at the cost of some additional off-policyness introduced from swapping in the old sample. Stored trajectories are periodically dropped when they become too old and would introduce too much off-policyness.
In addition to this we have also implemented adaptive task sampling that runs on the environment side before the learn step is entered. The environment holds the task dataset and reports how many rows it has, each group of trajectories is generated from one sample in this dataset. A row is drawn more often when recent groups on it have different rewards, and less often when they tie. This adaptive sampling increases the likelihood of reward variance in groups and therefore leads to less zero advantage scenarios and the discarding of computationally expensive rollout trajectories.
We have also implemented a plethora of optimizations in Arena’s RL engine to increase speed and stability, giving it the edge over open source alternatives:
The below chart shows how the trained models from all respective RL frameworks performed on the held-out eval dataset. Once again, it can be seen that Arena provided the greatest improvement compared to other RL libraries, beating the next best by 6.2 pts. In addition to this, we also benchmarked Claude Sonnet 5.5 and Opus 5.5 using the same harness, beating Sonnet by 21 pts and remarkably scoring within 1.3 points of Opus 5.5. After fine-tuning with Arena, our 35-billion-parameter model (3B active) is within 1.3 points of a model with parameters in the order of trillions.
There are two main takeaways from the data:
For a small up-front training cost, you own your model weights outright. You’re no longer exposed to per-token pricing that scales with your usage, or to a provider deprecating or changing the model your product depends on.
To demonstrate how the Arena-fine-tuned agent actually performed, the figure below takes us through a rollout our model completed during eval. This animation highlights the agent's thought processes, tool calling capabilities, and decision making to accurately answer the user’s query.
One of the toughest challenges businesses encounter when fine-tuning their own agent is the lack of data or simulation needed to see real performance benefits and while simulations such as Agent World Model are a positive step towards enabling smaller, specialized agents, there is still an enormous infrastructure gap when it comes to using RL to fine-tune an open-weight model. Environment orchestration, model distribution, async rollout and trainer engines all make up an incredibly intricate and complex web of systems that have to work seamlessly to see results that start to match the frontier.
This complexity is completely removed with Arena. Out of the box, Arena’s training and deployment stack just works. Get training on a dataset or environment in minutes, on your cloud or ours. Furthermore if your solution requires more custom data or a specific environment, we have engineers ready to work with you to close this data gap and to get you training your own models ASAP.
To start training your own tool-calling agent, sign up at arena.agilerl.com, or book a demo and we will walk you through it.