Snorkel Uses Modal Sandboxes to Scale Agent Evaluations and Cut Costs
- Gate
- Machine Learning
- Departs
- Crew
- Christopher Davis
- Passengers
- 913
Snorkel runs millions of agent simulations monthly with Modal’s CPU and GPU sandboxes, using memory bursting to improve reliability and reduce sandbox costs by about 20%.
Snorkel uses Modal to run large-scale agent evaluations, combining realistic execution environments with model inference. The setup supports millions of simulations each month, with demand reaching tens of thousands of sandboxes per hour. Snorkel says memory bursting has reduced its total sandbox bill by around 20% while helping teams avoid failures caused by tight resource limits.
For teams evaluating agents, the environment is part of the measurement. A coding agent, an infrastructure agent and a machine-learning agent may need different tools and compute. If a task fails because the sandbox lacks a GPU or the right system capabilities, the result says little about the agent itself. Snorkel therefore needs environments that can reproduce varied workloads, from multi-service systems to GPU-enabled model training.
Simulations need realistic infrastructure
Snorkel evaluates agents across coding, infrastructure, machine-learning engineering and other tasks. The company contributes to Terminal-Bench and runs both public and proprietary benchmarks against new and fine-tuned models. Its infrastructure has to support the tools each benchmark expects: an ML task may require GPU access, while an infrastructure task may depend on a full Linux kernel.
Modal’s sandboxes provide both CPU and GPU environments, including support for complex, production-like setups. Snorkel says GPU sandbox support at its required scale was a deciding factor. Using one provider for CPU and GPU workloads also makes its infrastructure and cost tracking simpler.
“For agent simulations to be meaningful, the environment needs to represent the real world the agent will operate in. Modal’s VM sandboxes let us recreate complex, production-like systems — from multi-container infrastructure workloads to GPU-enabled deep learning environments — across the millions of simulations we run each month.”
Evaluation speed matters because customers typically want an indication within a day of whether a model may suit their use case. Snorkel’s monthly volume can surge to tens of thousands of sandboxes in an hour as it tests newly released or tuned models. That workload makes high concurrency a practical requirement: the team needs to bring up many isolated runs quickly enough to finish evaluations on a customer’s timeline.
Elastic resources address unpredictable workloads
Before using bursting, Snorkel had to estimate how much CPU and memory each sandbox would need. Underestimates could cause resource-limit failures and invalidate a run. Overestimates left capacity unused and increased costs.
Those estimates are difficult because resource use can swing sharply during a task. A sandbox may sit idle while an agent waits for a model response, then consume more resources during a command such as a search through files. The workload is also hard to predict in advance because agents generate code as they work, and Snorkel runs a broad range of benchmarks.
Modal’s bursting lets a sandbox expand its CPU or memory resources when demand rises. Customers are charged for the resources in use at a given time or the originally requested amount, whichever is higher. Snorkel is still adopting the feature: roughly 10–15% of its sandboxes currently use bursting.
The change allows Snorkel to request fewer resources at setup while retaining room for temporary peaks. The company reports fewer runs failing for lack of resources, along with lower provisioning and about 20% savings on its total sandbox bill. That figure is Snorkel’s reported result; the source doesn’t break it down by workload or compare it with another provider.
“We need to support tasks from SMEs or agents, without over-provisioning infrastructure. Bursting gives us flexibility at configuration and runtime, avoiding over-provisioning and saving us around 20% on our total Sandbox bill.”
Inference supports changing evaluation needs
Running a simulation also requires access to the model being tested. Snorkel needs to start inference endpoints, run benchmarks and shut the endpoints down as evaluations change. For some initial model evaluations, an off-the-shelf endpoint is enough. Other tests require serving a customer’s fine-tuned model with custom weights.
Snorkel uses Modal’s shared and dedicated endpoints for those workloads. The team can use shared inference or load its own model without involving its infrastructure team, then scale endpoint capacity as benchmark demand changes. The arrangement brings model serving and sandbox execution into the same evaluation workflow.
“Modal’s serverless GPUs mean we can bring our own model or use shared inference, use the inference for different benchmarks, all while scaling really fast and reliably.”
Training is a possible next step
Snorkel says it has worked with Modal for roughly six months and has adopted new capabilities, including VM sandboxes, as they became available. Its current use centers on inference and simulation. The next area under consideration is training: Snorkel plans to evaluate Modal for model training, which could connect its data work with a cycle that runs from training through evaluation.
That remains a plan, rather than a reported deployment. For now, the practical case rests on the evaluation workload: realistic CPU and GPU environments, enough concurrency for rapid benchmark runs, flexible resource allocation and endpoints for both shared and custom models. As agent tasks continue to demand different system capabilities, the key measure will be whether this infrastructure keeps evaluations both representative and affordable.
