Enhancing Serverless Function Performance with a Geographically Distributed I/O Plane

Modal has introduced a game-changing update to its I/O system for serverless functions, effectively reducing end-to-end latencies by targeting specific geographic regions. This enhancement is especially relevant for applications that rely on quick feedback loops, like embedding models used in retrieval-augmented generation (RAG) setups.

Previously, function calls always routed through Modal's central I/O servers in the US East region, which could introduce latency upwards of 200 milliseconds for users located in Europe. The potential delay represented a considerable burden, especially for microservices or applications where low-latency interactions are crucial. The new design, however, allows developers to specify different regions for both their containers and the corresponding function's I/O routing, thus minimizing unnecessary network hops.

The New I/O Plane

Modal's newly architected I/O plane is now available in four geographic regions. This upgrade aims to optimize the input and output pathways of serverless functions, reducing latencies by as much as 80 milliseconds during p50 latency benchmarks. Users can simply set the routing_region function flag to choose the optimal route for their function calls. For instance, a function can be executed in the EU while still allowing I/O routes to work within the same region, speeding up processing for users in that area.

This shift not only speeds up function calls but has also streamlined the deployment experience for developers, especially those who had previously struggled with latency-induced bottlenecks when deploying services across regions.

Life of a Function Call

To understand how the new I/O plane functions, let’s examine the lifecycle of a function call. When a function is invoked, the client serializes the input and sends it to the I/O plane, which then queues it for processing. If any issues arise during processing—like server outages or code errors—the client will automatically retry until the function executes successfully. This architecture brings the reliability needed for production workloads, while the new server model efficiently manages resource allocation and scalability.

Moreover, async tasks such as updating autoscaling stats or publishing metrics now operate in the background, ensuring that primary function calls remain unaffected by non-essential processes.

Technical Improvements and Architecture

From a technical standpoint, Modal has rewritten its server layer using Go, favoring its concurrency model over Python’s async capabilities. This change aims to utilize lightweight threads that efficiently manage many simultaneous connections, significantly decreasing input processing times. While Google’s gRPC framework facilitates fast communication between functions, transitioning to Go's underlying framework has been pivotal in optimizing performance for heavily trafficked serverless compute tasks.

Additionally, Redis is leveraged for input queues, where Modal has adopted version 7.1 to avoid performance issues that surfaced in the latest releases. This careful choice showcases Modal's commitment to maintaining high performance under load, which is critical when dealing with real-time requirements.

Best Practices for Users

To make the most out of Modal's serverless functions, developers should be mindful of several effective strategies. Start by managing execution times judiciously to cut down on overall latency. If your user base is concentrated in a specific area, choose a routing region that minimizes the distance data must travel.

For larger inputs exceeding 2MB, the idiosyncratic approach requires uploading to blob storage first, which can add latency. Keeping payload sizes below this threshold can lead to improved performance. Lastly, batching multiple smaller inputs is another effective technique. By aggregating these inputs into larger requests, the system can avoid multiple network hops, thereby increasing efficiency.

Conclusion

The introduction of a geographically distributed I/O plane fundamentally enhances how Modal Functions operate, providing developers with improved reliability and performance. Through rigorous testing and architectural overhauls, Modal has ensured this system will serve as a strong foundation for future innovations in serverless computing. As the platform evolves, continued focus on user experiences will drive further improvements, reinforcing Modal's position in the serverless ecosystem.

Special thanks to the Core Services I/O team for their tireless efforts in implementing this significant migration. The successful launch of these optimizations exemplifies the commitment to enhancing user satisfaction in every aspect.