NVIDIA L40S GPUs Launch with Enhanced Performance and New Features on Modal

🚀 Introduction of L40S GPUs

NVIDIA has launched its L40S GPUs on the Modal platform, available for $1.95 per hour. These GPUs come with a remarkable 48GB of GDDR6 RAM and promise improvements over the widely used A10 GPUs.

• The L40S GPUs provide double the memory of the A10 models, allowing for the execution of larger AI models and accommodating longer context windows.
• For memory-bound tasks, the L40S achieves up to a 40% increase in processing speed.
• Compute-bound jobs benefit significantly from utilizing 16-bit Tensor Cores, achieving speed enhancements exceeding 100% compared to prior architectures.

To utilize this new GPU in your applications, simply include the following decorator in your function:

@app.function(gpu="L40S")

🔒 Introduction of Proxy Auth Tokens

Modal has augmented its capabilities with the introduction of Proxy Auth tokens, which facilitate authenticated access to web endpoints. This enhancement helps developers control access, ensuring unauthorized users do not incur unnecessary charges.

@app.function(gpu="h100")
@modal.web_endpoint(requires_proxy_auth=True, docs=False)
def expensive_secret():
    return "I didn't care for 'The Godfather'. It insists upon itself."

📷 New Filesystem API and Disk Snapshotting for Sandboxes

The newly implemented Filesystem API allows users to easily read and write files within their Sandboxes, promoting a smoother workflow for file management during interactive sessions.

sb = modal.Sandbox.create(app=app)

filepath = "/home/foo.bin"
with sb.open(filepath, "wb") as f:
    f.write(b"Hello, World!")

f = sb.open(filepath)
print(f.read())

In addition, the introduction of disk snapshotting for Sandboxes improves the overall efficiency of session management. Key capabilities include:

1. Creating complete snapshots of a Sandbox's state for future reference.
2. Branching from these snapshots to spin up new Sandbox instances with modified configurations.
3. Reducing cold-start delays by restoring environments directly from snapshots, allowing for quicker response times.

👩‍💻 Client Updates

To access the latest features, ensure to run pip install --upgrade modal. Here are the highlights of the recent client updates:

• Image Management: Modifications to Image.from_dockerfile() and image.dockerfile_commands() have streamlined image processing by automatically locating and utilizing a .dockerignore file.
• File Matching: The FilePatternMatcher now offers a from_file constructor, enabling file pattern definitions to be read directly from external files instead of needing to be specified inline.
• Volume Management: It's now possible to rename Modal Volumes through the CLI with modal volume rename or the SDK with modal.Volume.rename.
• Sandbox Enhancements: Filesystem notifications resembling fsnotify are now supported in Sandboxes, and the maximum write payload has increased to 1 GiB.
• Environment Configuration: The App.run context manager introduces a new environment_name parameter for finer environmental controls.
• VSCode Integration: Users can now launch VSCode using an arbitrary Dockerhub base image with the command:

modal launch vscode --image=nvidia/cuda:12.4.0-devel-ubuntu22.04

🔐 SOC 2 Type 2 Certification

Modal also announced its successful SOC 2 Type 2 certification, reinforcing its commitment to security and operational effectiveness. Interested parties can request the report or further inquiries at [email protected].

📚 GPU Glossary

To aid developers navigating the complexities of GPU technologies, Modal has compiled a GPU Glossary. This resource includes concise definitions and high-quality materials covering topics such as Tensor Cores, Warp Schedulers, and CUDA Toolkit functionalities.

🧬 New Computational Bio, OCR, and Image Diffusion Resources

• ESM3: A state-of-the-art model for predicting and generating protein structures from sequences. A comprehensive demonstration can be found in the Protein folding dashboard example on Modal.
• GOT: A powerful 580M parameter OCR model proficient in managing diverse content formats. Check out its capabilities through our Example on Modal.
• Recently, Modal hosted a webinar covering best practices in productionizing diffusion models. You can watch the complete session here.

🍭 Fun Tidbits

• Exclusive dinners for biotech founders and engineers are being organized. Interested individuals are encouraged to reach out for participation details.