Legora Runs 2,000 Parallel Agent Evaluations in Modal Sandboxes

Legora’s move from an AI assistant to a long-running agent changed what its evaluation system had to prove. The agent works across research, review and drafting tasks that can span thousands of documents, so small tests with a handful of files no longer reflected the product it would encounter. Legora rebuilt its evaluation infrastructure around isolated copies of the full application, running thousands of experiments at once in Modal Sandboxes.

The scale is substantial: a typical experiment combines 100 task cases, four harness variants and five trials per case. That produces 2,000 runs, each needing its own environment. The setup lets engineers start experiments from their laptops while the work runs concurrently in the cloud.

Testing the whole application

Legora’s earlier evaluations could run locally because the assistant handled smaller tasks in environments with only a few documents. The agent can run for hours in a project containing thousands of documents and gigabytes of data. Scaling the number of runs was one challenge; making each run representative was harder.

The agent uses many parts of Legora’s product: a headless DOCX editor, self-hosted Turbopuffer instances, PDF conversion, Postgres and authorization databases, blob storage, and code execution environments created as needed. A test that substitutes mocks for these components could miss the effects of a code change or database migration. Legora therefore boots the full application, pinned to a specific commit, in each evaluation environment.

That design makes each run more faithful to product conditions, while also raising the cost of setup. Large application images take time to build, and the test data must be restored consistently. Modal’s managed Sandboxes gave Legora a way to allocate resources for the application and run environments in parallel, with startup speed and support for complex setups among the reasons the team selected the service.

Sharing data while keeping runs repeatable

Each Sandbox receives a custom image and filesystem. Modal Volumes store large document collections and let the team snapshot the state needed to begin a run. That shared setup includes documents, database state, permissions and blobs, so repeated trials can start from the same conditions without duplicating every part of the preparation.

Legora also placed cached Docker images in Volumes to shorten the Docker Compose setup. The team says it plans to move that cache to filesystem snapshots. The distinction matters in a workload with thousands of environments: reducing repeated setup can lower the time and storage required for a large experiment, while preserving the same starting state across trials.

Building the full application image initially slowed evaluations. Legora now runs an hourly pipeline to build the image and pre-warm Sandboxes. When an evaluation needs a particular commit, the reset process can apply the changes since the previous build instead of rebuilding the whole application. Another hourly job copies secret values from Legora’s key vault into Modal Secrets. The team also describes frequent secret rotation, least-privilege access, dependency-age requirements and domain allowlisting as parts of its environment security.

Running Docker inside the Sandbox

Legora first used Sandboxes backed by gVisor, then adopted Modal VM Sandboxes as an early user. Each VM Sandbox includes a Linux kernel, allowing Docker to behave more like it does on a standard Linux host. That removed several workarounds the team had needed for networking, DNS resolution and FUSE mounts. Previously, those constraints also led it to run code sandboxes as bare processes; with VM Sandboxes, Legora says it removed those adjustments.

This is a practical infrastructure choice, not just a matter of convenience. Legora’s application is composed of Docker services, so the closer the test environment behaves to the deployment environment, the more directly an evaluation can measure changes across the product. The tradeoff is that each run carries the requirements of a large application, which makes image preparation, startup time and resource allocation central parts of the system.

Using evaluations to improve the agent

The platform now underpins BAR, Legora’s internal benchmark of realistic, long-horizon legal tasks. The company uses it to compare foundation models on quality, cost and latency. Those measurements feed into production model selection and choices about the agent harness—the tools and procedures that shape how the agent completes a task.

Legora has also made the evaluation system accessible through a command-line interface and a skill, enabling agents to create cases, launch experiments and analyze results. The company says most engineers now interact with evaluations through agents. Given a metric, a set of experiment tools and a budget, an agent can test harness changes and iterate toward a better result.

The team has applied that process to tool search, PowerPoint generation and reproducing research papers. In one reported example, changes to compaction and a custom tool search implementation reduced agent costs by 30%. Agents helped validate and improve the results through automated research, while engineers oversaw the work.

For teams building agents that operate across full applications, Legora’s account points to a clear evaluation requirement: tests need to reproduce the application state the agent will actually use, and they need enough parallel capacity to make comparisons practical. The next question is how reliably these benchmark gains carry into production tasks, where document collections and user workflows may differ from the cases in the test suite. Legora’s infrastructure makes those comparisons possible at scale; the quality of the benchmark still determines what the results mean.

Legora’s account of its evaluation system is available in the company’s engineering post.