Strategies for Maintaining GPU Reliability in a Large-Scale Cloud Environment

Modal operates an extensive, globally distributed GPU worker pool, leveraging resources from multiple cloud providers such as AWS, GCP, Azure, and OCI. With over 20,000 concurrent GPUs and more than four million cloud instances launched in recent years, Modal has encountered a wide variety of GPU reliability challenges. This article details Modal's GPU reliability system and serves as a guide for organizations utilizing cloud services.

Instance Type Testing and Selection

Understanding the reliability of different cloud instance types is crucial. Modal's analysis reveals significant performance and reliability variations among providers. For instance:

• Cloud A: Features a straightforward and dependable instance launch API. When requesting a bare metal (BM) or virtual machine (VM), there's a 99.6% success rate of booting within 2-3 minutes.
• Cloud C: Experienced overheating H100s, sometimes peaking at over 90ºC. Given that FLOP/s performance drops in temperatures above mid-70s Celsius, this has severe implications for sustained workloads.
• Cloud D: Provides strong price/performance metrics, but has reported issues like hardware clock slowdowns and uncorrectable ECC errors, making it imperative to understand the instance type thoroughly.

To navigate these differences, Modal employs a semi-automated benchmarking tool called modal-host-bench that evaluates performance and reliability issues. The benchmarking data illustrates that certain GPU models (SXM H100) drastically outperform others (PCIe H100), affecting the choice of instances for user workloads.

Machine Images

The quality of machine images used for bare-metal and virtual machine servers heavily influences reliability and performance. These images, comprised of the kernel, OS files, NVIDIA drivers, and necessary configuration, must be consistent and up-to-date across Modal's multi-cloud environment.

In the earlier stages, Modal's image updates were performed manually, which led to frequent errors. The process evolved to a continuous integration model with automated testing for reliability and performance checks before any new image was deployed.

Another critical consideration is the ability of cloud providers to load custom machine images. Most neocloud services lack the support for such capabilities, resulting in inferior instance startup performance. For instance, Cloud C boasts rapid boot times, averaging under two minutes, while many neocloud services struggle to boot within five minutes.

Instance Boot

Once a machine image is installed, the next challenge is ensuring successful instance booting. If an instance is booted with faulty GPUs or if the cloud-init process has errors, timely detection is essential to prevent customer impact.

Modal's autoscaling fleet requires that startup processes be efficient; any increase in startup latency can hinder failover reliability. Therefore, the checks performed during booting are deliberately light to strike a balance between reliability and operational efficiency.

Error checking during boot typically includes queries to systemctl and nvidia-smi, along with a basic read-write test on a randomly selected GPU. Modal has managed to maintain an extremely low failure rate during this phase, with only a small occurrence of issues attributed to specific GPU types from certain cloud providers.

Lifetime Management

After successful boot, continuous monitoring is required to maintain a healthy operational state. Modal implements both passive and active health checks throughout an instance's lifecycle. Passive checks include monitoring logs and metrics without engaging the GPU, while active checks require locking the GPU for diagnostic tasks.

Passive Healthchecking

Modal finds that 80% of health monitoring can be achieved with minimal effort, using tools like dcgmi and system logs to monitor uncorrectible ECC errors, thermal issues, and excessive temperatures.

Active Healthchecking

In contrast, active health checks are more resource-intensive but necessary to validate GPU resilience under load. Modal ensures each GPU undergoes comprehensive active checks weekly, including NVIDIA's diagnostic tools and performance benchmarks. If any checks fail, the instance is quarantined for further examination.

Future plans include implementing network-oriented checks due to rising interest in high-speed interconnects, crucial for training and inference tasks.

Observability

Through a dedicated dashboard, Modal provides users visual access to GPU health indicators. Key metrics include memory usage, GPU utilization, temperature, and power consumption. While clear at a container level, deeper insights require aggregating data from multiple containers.

Abnormal health events are logged and provided in the dashboard logs, allowing users to quickly understand potential GPU issues affecting their workloads.

Support

Despite strong measures in place, edge cases can arise, necessitating a reliable support system. Modal offers a dedicated support channel for enterprise clients, facilitating quick resolutions. For broader community support, the company maintains responsiveness and provisions credits for identified GPU failures that impact users.

Conclusions

GPU unreliability is an ongoing challenge that can impede AI and machine learning workflows. Understanding the issues that can arise with GPU hardware is paramount for teams deploying these technologies. As seen in various studies, the challenges of GPU failures often far exceed those encountered with CPU failures.

Until systems improve to achieve consistent reliability equivalent to CPUs, Modal remains committed to providing robust GPU performance monitoring and support to ensure that customers can confidently leverage GPU power.