Modal Clusters: A New Approach to Scalable Multi-Node Compute for AI Training and Inference

Organizations today face a significant challenge: owning the intelligence necessary for success while managing the vast computational resources required for training and deploying AI models. Modal Clusters, introduced after 1.5 years of extensive testing, provide an innovative solution for scaling up AI workloads without the need for heavy investments in hardware.

Launched as a generally available offering, Modal Clusters can be easily utilized with a decorator, @modal.clustered. This simple command allows developers to define groups of nodes that work together for intensive computing tasks. Here's an example code snippet to illustrate:

@app.function(gpu="B300:8")
@modal.clustered(size=4, rdma=True)
def train_model():
    cluster = modal.Cluster.from_context()
    container_ips = cluster.container_ips()
    container_rank = cluster.container_rank()
    
    world_size = len(container_ips)
    main_addr = container_ips[0]

    print(f"{container_rank=} {world_size=} {main_addr=}")
    ...

Modal Clusters seamlessly integrate with pre-existing Modal components such as Volumes, Cloud Bucket Mounts, and Queues. Additional benefits of Modal Clusters include:

• High throughput communication: Nodes communicate via InfiniBand verbs with a speed of up to 6.4 Tbps, automatically optimized for frameworks like PyTorch and NCCL.
• Rapid cluster initiation: Your code can run on a cluster within seconds, billing is done by the second, making it cost-effective.
• Comprehensive GPU health assurance: Modal’s automated health checks and remediation capabilities now extend to RDMA (Remote Direct Memory Access) health.

Moreover, with Modal's on-demand cluster access, users have the flexibility to run jobs whenever they need and only pay for the resources they use, contrasting with traditional mechanisms that bill by the hour or require reservations.

Reimagining the Modal Stack

Introducing clusters necessitated a complete overhaul of Modal’s systems. The traditional single-node scheduler was insufficient for managing multi-node workloads, prompting the development of a new gang scheduler.

Gang Scheduling: Coordinating Multiple Nodes

The gang scheduler operates on an advanced observe, plan, and act loop, which allows it to evaluate the fleet and determine optimal placements for clusters simultaneously. It functions as follows:

1. Collects metadata for all active multi-node workloads.
2. Assigns workloads to nodes according to availability, organizing by Availability Zone (AZ) and network ID.
3. Provision new capacity for workloads that don't currently fit.
4. Notifies all active nodes of the assigned workloads.

This holistic scheduling approach minimizes resource fragmentation and optimizes the utilization of Modal's compute pool, enabling fast acquisition of multi-node clusters.

RDMA Support

Multi-node workloads typically handle substantial amounts of data. For instance, a training step in the GLM 4.7 model, which synchronizes approximately 717 GB of BF16 weights, is performed significantly faster using RDMA compared to traditional TCP connections. RDMA allows for direct memory access between hosts without routing through kernel space, reaching speeds of up to 6.4 Tbps.

However, integrating RDMA has its complexities due to varying implementations across cloud providers. Modal addressed this by allowing users to enable RDMA with a single flag: rdma=True. This capability simplifies setup, ensuring that workloads utilizing PyTorch with NCCL function out-of-the-box without needing to navigate the intricacies of RDMA configurations.

The following capabilities are automatically configured when rdma=True is set:

• Setup of RDMA fabric reaching up to 6.4 Tbps per node.
• Automatic driver installation and NIC configuration across the hardware.

Modal has also improved its secure multi-tenant compute environment by integrating RDMA into the gVisor container runtime, enhancing both performance and security.

Real-World Applications of Modal Clusters

Several organizations are harnessing the power of Modal Clusters to push forward their AI capabilities:

Decagon: Enhancing Customer Experience

Decagon leverages Modal Clusters to train AI agents aimed at improving customer interactions. They have succeeded in fine-tuning models with up to a trillion parameters using Miles, focusing on quality and efficiency in response times. Harnessing Multi-node capabilities, Decagon is able to sharpen the performance of their agents, as highlighted by their Research Lead, Cyrus Asgari:

“Model performance is central to the customer experience at Decagon. Modal Clusters helped us train at scale, allowing us to focus on enhancing our agents.”

1x: Building Intelligent Robots

1x is developing NEO, a home robot powered by a sophisticated world model pre-trained on Modal Clusters. Their testing requires flexible workloads, and the integrated Modal scheduler allows them to efficiently manage both large-scale runs and smaller evaluations:

“Modal Clusters let us scale up to hundreds of B300s with RDMA efficiently; we only pay for what we use.”

Runway: Elevating Video Generation Models

Runway has harnessed Modal Clusters to power their new video generation models, producing high-quality output in seconds across multiple nodes. This utilization emphasizes the challenges of scaling multi-node inference, and the benefits of Modal’s autoscaling features:

“Modal’s platform has aided in rapidly scaling our production inference workloads to meet increasing demands.”

Getting Started with Modal Clusters

Modal Clusters are available for use today. Developers can begin utilizing this powerful resource for AI workloads with just a line of code. The cluster size is limited by your account's GPU limits, so for ambitious projects, it's advisable to reach out for assistance.

To start utilizing Modal:

1. Install Modal: pip install modal
2. Create an account: python -m modal setup
3. Consult the Multi-node Clusters Documentation for further guidance.

With these tools at your disposal, the potential for innovative application development is immense.