Modal Batch Launches with Enhanced Job Processing Capabilities for AI Applications

Modal has announced the launch of Modal Batch, a new interface that leverages a durable queue system designed for scalable and fault-tolerant job processing. This innovation comes as a response to the growing demands from AI customers who require efficient batch processing solutions. With tasks ranging from audio preprocessing to document embedding, users need an effective way to manage substantial data loads simultaneously. The ultra-fast container infrastructure offered by Modal makes it particularly adept for these use cases, with firms like Harvey, Suno, and Achira already utilizing the Modal Batch capabilities.

How It Works

To illustrate the functionality of Modal Batch, consider the scenario where a user needs to embed one million video clips. The first step is to define a Modal Function dedicated to performing the embedding, with options to specify custom images or particular hardware if required. An example code snippet shows how to configure this:

import modal

app = modal.App("video-embedding")

@app.function(
    gpu="L40S",                     # GPU-accelerated processing
    retries=2,                      # Handle transient errors
    volumes={"/videos": volume},    # Mount a distributed filesystem
)
def embed(video):
    # Business logic!
    ...

After defining the function, the user can call .spawn_map with their list of video clips as inputs. This command is designed to launch thousands of containers in the cloud, ensuring a reliable execution for all inputs. Here’s a snippet demonstrating this process:

@app.local_entrypoint()
def main():
    embed.spawn_map(videos)

Successfully running this setup showcases Modal Batch’s ability to spin up over 100 containers to process 25,000 inputs concurrently.

Improvements Over Previous Options

While batch processing was previously possible on Modal, the launch of Modal Batch incorporates several critical enhancements. Improvements include:

• An overhaul of the queue system that now accommodates 500 times more inputs, allowing for one million inputs to be queued simultaneously.
• A guarantee that inputs will be executed up to 7 days after initiating a batch job, expanding the window from 1 day.
• The introduction of a native .spawn_map handler that simplifies the spawning of batch inputs without the need for asynchronous logic management.

Why Choose Modal Batch?

Setting up batch processing usually involves navigating complex distributed systems for handling payload storage, batch orchestration, cloud resource scaling, and monitoring. Modal Batch capitalizes on Modal’s strengths, allowing users to concentrate on their business logic instead of infrastructure management.

Enterprise-grade scalability: Modal’s custom container runtime, filesystem, and scheduling mechanism enable rapid scaling. Users can instantly deploy thousands of containers, including GPUs, without needing any reservations.

Ergonomic user experience: There's no need to exit the development environment to launch a batch job; Modal Batch eliminates the dependencies on cloud consoles, YAML configurations, or container orchestration systems.

Ease of debugging: With Modal's dashboard, users can access logs and metrics for individual inputs and containers, facilitating quick identification of failures, with aggregate metrics available for overall batch performance.

Integration with existing ML pipelines: As a general-purpose compute platform, Modal allows for easy chaining of Functions across batch processing, model training, or inference, leveraging the same Python interface.

Customer Use Cases

Since Modal Batch entered beta, customers have successfully developed a range of applications. Here are key examples:

Document Processing at Harvey

Harvey is an AI platform that streamlines document workflows for legal teams. Their requirement for quickly processing millions of documents for insights led them to use Modal Batch, resulting in a tenfold increase in processing speed. Previously reliant on open-source orchestration tools like Argo and Airflow, Harvey faced challenges managing infrastructure expenses and scalability.

Audio Processing at Suno

Suno, a leader in AI music generation, depends on Modal to preprocess vast datasets essential for training audio models. With the capability of their 4.5 model to generate up to eight minutes of high-fidelity audio at 48kHz, the data processing demands are significant. Using Modal Batch allows Suno to:

• Launch GPU-heavy batch jobs immediately without prior infrastructure setup.
• Utilize GPU-accelerated models for high-bandwidth audio data.
• Maintain minimal script complexity, allowing workflows that reliably “just work.”

Processing Scientific Datasets at Achira

Achira focuses on advancing drug discovery with atomistic simulation models, which require validating extensive quantum mechanical datasets. Modal Batch's robust retry mechanism and batching capabilities have significantly enhanced Achira’s data pipeline reliability.

Future Developments

Modal is committed to extending its batch processing functionalities while ensuring ease of use remains a priority. Upcoming features include:

• Function Caching: Prevent redundant computations.
• Improved Ergonomics: Programmatic interactions for job management—changing priorities, checking progress, and cancelling jobs directly from Python code.
• Multilingual SDK Support: Enable batch jobs via SDKs in languages like JavaScript.

Getting Started

For developers ready to engage with Modal Batch and streamline batch job processing:

1. Install Modal: pip install modal
2. Create an account: python -m modal setup
3. Explore the batch processing documentation

Join the Slack community for support or to share feedback. The team is eager to see what new applications users will create!