Enhanced Batch Processing and New Inference Example with Modal Updates

Scaling Batch Job Processing

Modal has significantly upgraded its capabilities for running large-scale asynchronous jobs. This enhancement allows users to now queue up to 1 million inputs per Modal Function, a substantial increase from the previous limit of 2,000. This is a major improvement for teams grappling with extensive datasets or those needing complex computations. Large-scale jobs aren't just a fancier feature—they're essential for analytics, machine learning training, and real-time data processing. Now, organizations can optimize their workflows, achieving efficiency without compromising on scale.

Additionally, Modal has increased the .spawn() rate limit, permitting quicker submissions of input to functions. For developers, faster submissions mean reduced latency and more immediate feedback, which is vital for tasks like iterative testing or live data validation. The implications here stretch beyond simple speed; they touch on time-to-market advantages for application updates and reduced friction in collaborative environments. This could notably enhance workflows where timing is crucial, such as in finance or real-time analytics platforms.

In line with job management enhancements, FunctionCall results will now persist for a duration of 7 days, an upgrade that ensures results can be retrieved without the pressure of immediate processing completion. This shift not only offers a practical buffer for developers but also underscores a growing trend towards more user-centric job management approaches, enabling teams to reactivate or examine past jobs without losing track of critical outputs. The change may seem small, but it reflects a broader industry recognition of the need for flexibility in asynchronous computing.

For those keen on implementing these job processing features, Modal offers a detailed guide available here. This resource will be essential for developers transitioning to the new systems, allowing them to leverage the full potential of these updates.

Client Updates and New Features

Users should run pip install --upgrade modal to access the latest client updates, which are noteworthy. The upcoming Modal Client v1.0 introduces a cleaner API structure, which should enhance usability and reduce the learning curve for newcomers. However, there are some deprecation warnings to heed. Developers might find some features obsolete in this new environment, so it's crucial to consult the Migration Guide for adequate preparation.

Another significant addition is the ability to launch ephemeral apps from containers using with app.run():. This capability offers developers new possibilities for temporary applications that can spin up quickly for testing or demonstration. Yet, caution is advised; it's recommended to keep this functionality out of the global scope to avoid recursion issues. The last thing any developer wants is a runaway process in their code.

The introduction of context_dir enhances the reliability of relative COPY commands in Dockerfiles, making deployment smoother. By establishing clearer paths for resource management, Modal responds directly to major pain points that developers have highlighted in the past. Moreover, you can now define default entrypoint arguments for Docker images using Image.cmd(...), further streamlining the deployment process. Together, these updates indicate that Modal is listening to user feedback and gradually iterating to improve the development experience.

Tracking development has been bolstered too, allowing developers to view Git commit information for apps directly through modal app history in the CLI and in the dashboard. This creates a more cohesive development experience, making it easier to keep tabs on changes and improvements over time.

TensorRT-LLM: Fast LLM Inference

Modal has released a new example highlighting how to serve large language models (LLMs) with TensorRT-LLM, achieving ultra-low latency of less than 400 milliseconds. This is increasingly important as businesses lean on LLMs for interactive applications, where delayed responses can hinder user experience. If you’re in a market where real-time feedback is a key selling point, the implications of this speed cannot be overstated. The example can be explored here, perfectly illustrating practical scenarios for teams looking to roll out advanced language models.

Video Walkthroughs for Enhanced Learning

To assist users in effectively utilizing its offerings, Modal has released two comprehensive walkthrough videos that could be of great benefit:

• Deploy DeepSeek models on Modal — A detailed step-by-step guide for spinning up DeepSeek in production; you can watch it here.
• Serve OpenAI-compatible APIs with vLLM — This instructional video covers how to deploy and scale a fast vLLM service on Modal; you can view it here.

Recent Customer Launches

• Imbue has unveiled Sculptor, the first coding agent environment designed to assist developers in catching issues, writing tests, and improving code quality using Modal Sandboxes. Such tools are becoming staple offerings that help teams ensure higher code integrity.
• Phonic launched its new voice AI platform, leveraging Modal for low-latency inference and massively parallel job processing that could redefine user interaction across various media and applications.
• Firebender has introduced Kotlin-bench, which benchmarks AI models on real-world Kotlin and Android tasks, employing Modal’s .map() for efficient parallelization. Here, the ability to handle multiple jobs at once is particularly beneficial.

Fun Tidbits from Modal

• Modal was recognized as the #2 most promising early-stage company on the 2025 Enterprise Tech 30 list by Wing VC and Eric Newcomer, which speaks volumes about its potential and recognition in the tech community.
• The team hosted a successful open-source LLM demo night in collaboration with Mistral, showcasing innovations like fast speech-to-speech systems—an exciting glimpse into how far speech technology has progressed.
• Modal launched its first billboard campaign in San Francisco, offering prizes to individuals who share photos of their advertisements on social media. (and this is the part most people overlook) Such marketing efforts often prove effective in building brand awareness in the tech sector.

Future Outlook for Modal

As Modal continues to enhance its infrastructure and user experience, several exciting prospects arise. The improvements in batch job processing and real-time function capabilities are likely to attract a broader user base, particularly among developers who require high-volume data handling. This shift may position Modal as a serious competitor in cloud computing and data processing niches.

Moreover, features like TensorRT-LLM not only push the boundaries of what’s currently possible with LLMs but also set a new standard in latency-sensitive applications. Companies that align theirs with these advancements may gain an edge in responsiveness, customer satisfaction, and operational efficiency.

As more organizations seek to implement machine learning and AI into their workflows, platforms that can accommodate both beginner developers and seasoned experts will rise to prominence. Modal seems poised to capitalize on this demand, but the ongoing challenge will involve staying ahead of the curve while managing user expectations for seamless integration and performance.

If you're working in this space, now's an opportune moment to explore what Modal can offer—these improvements signal a commitment to innovation, but the onus is on each developer to fully exploit what's available.