NVIDIA Vera CPUs Deliver Up to 2× Sandbox Throughput Under Load

Modal’s early tests suggest NVIDIA’s Vera CPU could let its Sandbox service run more concurrent work without the latency penalty that usually comes with packing workloads tightly onto a host. The company reports up to twice the task throughput of an equivalent x86 GNR machine, while individual task latency stayed similar or sometimes fell. That result matters because Sandbox workloads often compete for resources in unpredictable bursts.

Modal received its first Vera server at its San Francisco office. Vera is an Arm compatible CPU designed for NVIDIA’s Vera Rubin GPU platform and also offered in a standalone server. Supporting it would require changes across Modal’s infrastructure and image build system, which has historically run only x86 machines. The potential payoff is a different balance between host utilization and performance for Modal Sandboxes.

Why sandbox workloads are difficult to schedule

Modal runs hundreds of containers and virtual machines on each host. Each sandbox can behave differently: it may sit mostly idle while waiting for a model, then abruptly consume far more CPU during a command such as npm install or an expensive grep. The workload’s shape may not be known until runtime, so resource sizing is often an estimate.

Those workloads share a pool of compute despite their varied demands. When one tenant’s burst overlaps with another’s work, contention can affect both. Modal says that makes it hard for customers to size workloads and for its scheduler to place them without performance conflicts.

Modal supports bursting, so customers can reserve capacity around their P50 usage rather than paying for the worst case. A transient spike can draw on spare capacity within the host. The company says this approach has saved users millions of dollars, but it also means workloads can push into capacity that strict limits would leave unused. That makes a host’s ability to tolerate contention particularly relevant.

CPU speed affects more than CPU costs

Sandboxes often support a larger process, so CPU delays can slow work beyond the sandbox itself. Modal describes two common settings. Background agents, including those used by Ramp Inspect and Niteshift, can return answers sooner when their tasks finish faster; shorter execution can also reduce CPU time and operating costs.

Reinforcement learning rollouts present a different link. GPUs may wait while sandboxed processes complete their work. Many training setups run synchronously, so the next step cannot begin until a group of rollouts finishes. In that setup, saving time on CPU can also reduce idle time for the GPU system.

A benchmark built around mixed tasks

To compare processors, Modal created Agentless Agent Bench, a collection of 50 Terminal-Bench oracle solutions and verifiers. The tasks run in concurrent, prewarmed Docker containers, with network calls removed. This isolates CPU work agents and reinforcement learning systems may perform, including compilation, file searches, Git operations and device emulation.

Modal designed the benchmark to represent a concurrent mix of tasks rather than optimize for one resource, such as disk input and output or SIMD processing. It measures both individual task completion time and concurrent throughput. The first comparison ran selected tasks serially on an otherwise idle machine, showing straightline latency across NVIDIA Vera, Intel GNR, AWS Graviton 4, AWS Graviton 5 and AMD Zen 5.

Across the 50 tasks, Vera often finished ahead of the alternatives, but its lead was not consistent or especially large in the quiet-machine tests. Modal says those measurements don’t reflect the experience of a sandbox host, where many workloads compete for resources at once.

Throughput under contention changes the comparison

Latency measures how long one task takes. Throughput counts how many tasks finish in a given period. Packing more work onto a host can raise throughput, but when the CPU is saturated, contention often stretches each task’s runtime. For a scheduler, running hosts densely can increase capacity, while also risking longer waits when demand spikes across the cluster.

Agentless Agent Bench tests this by increasing the number of concurrent containers and counting completed tasks over a fixed interval. In Modal’s early runs, Vera’s throughput rose to as much as twice that of an equivalent x86 GNR machine. Individual task latency remained comparable and sometimes lower. Modal says Vera maintained its pace as jobs accumulated, while competing systems reached throughput plateaus earlier.

Memory bandwidth and the path to deployment

Modal attributes Vera’s behavior under load to high memory bandwidth. Its account is that this lets the host fit more container workloads per core without CPU throttling, helping explain the wider gap in concurrent tests than in serial ones. As Modal increased CPU counts in the comparison, alternatives plateaued earlier while Vera continued to gain throughput. The company suggests the results might improve further on a larger node, though its test was limited to the server available for benchmarking.

For Modal, the result could support denser host packing and more room to absorb workload bursts. For users, that could mean greater available scale and throughput while keeping task latency in check. These are early benchmark findings, and supporting Arm will require changes to Modal’s infrastructure and image builds. The next practical measure will be how the result holds across a larger fleet and real sandbox workloads as Modal works to expand Vera support with NVIDIA and cloud providers.