Modal Introduces Serverless Servers for Ultra-Low-Latency Applications

Modal has expanded its capabilities, enabling the execution of high-performance code in the cloud, including Python functions, agent runtimes, notebooks, batch jobs, and now, ultra-low-latency Servers optimized for HTTP, WebSocket, and gRPC traffic. These new offerings are designed for applications where every millisecond is crucial, such as large language model (LLM) inference for interactive agents.

Unlike the previous Modal Web Functions which were robust but designed with I/O processing—such as queueing and platform-managed request lifecycles—this new server model emphasizes minimizing latency in the core path without sacrificing essential features. The enhancement of server architecture allows Modal to achieve a commendable reduction of p50 latency from 39ms to just 6ms, thus making it more competitive in scenarios where latency is a critical factor.

What are Modal Servers Used For?

Modal Servers act as intermediaries, connecting clients to a regionalized, autoscaling pool of HTTP server replicas using a reverse proxy routing system. This architecture differentiates from Modal Web Functions as it focuses on instance management and performance optimization specifically in low-latency scenarios.

Servers allow a seamless integration of client requests with an intelligent routing layer that scales based on demand. It is essential for developers who need to maintain low-latency connections for real-time applications. The system supports various protocols, which can efficiently route different types of traffic to the correct backend services based on their requirements.

The architecture employs a lightweight routing mechanism, contrasting it with the previous heavyweight methods utilized by Modal Web Functions. This minimalist design approach is pivotal in achieving the significantly reduced latency required for time-sensitive applications like LLM inference.

Design Principles Behind Modal Servers

Every architectural choice made in building Modal Servers was guided by two core principles:

1. Maximize resource sharing while minimizing interference: This necessitates pooling resources across different tenants and requests to enhance performance while ensuring that specific requests experience minimal interference, preserving performance integrity.
2. Avoid network calls on the request path: By eliminating metadata fetches, key-value store interactions, or reliance on blob storage within the critical execution path, the design prevents latency issues that could arise from additional queueing or retry mechanisms.

The primary focus on resource sharing helps to balance the composite performance across users while retaining the illusion of dedicated resources ensuring reliability and speed. The avoidance of network calls limits potential bottlenecks during request handling, which is strategically essential for maintaining the lower latencies required in modern applications.

How Modal Servers Operate

The implemented routing system within Modal Servers consists of various components including a streaming edge proxy and intelligent stateless proxy, both designed to handle user requests efficiently:

The overall architecture is build around the AWS Elastic Kubernetes Service (EKS), although user containers are managed by a distinct scheduling system that is optimized for variable workloads across multiple cloud instances worldwide.

When clients make requests, they interact with the system through designated URLs that communicate with the appropriate Modal Server, establishing connections that utilize lightweight protocols to reduce overhead and enhance performance.

Life of a Request

The journey begins with a resource request identified by a URL structured as follows:

This URL component identifies which Modal Server to connect to, and the routing region specifies where the request should be processed, tapping into the autoscaling proxy system to handle incoming traffic efficiently.

Requests are handled through an AWS Network Load Balancer (NLB), which communicates with an L4 proxy optimized for TCP traffic. Subsequent processes involve Envoy, a powerful L7 proxy, that normalizes the traffic and dynamically routes it in HTTP/2 format to improve performance via multiplexing.

Latency and Scalability Improvements

Through innovative architectures, Modal has enabled responses from the server to clients to be completed in just 5-7ms, highlighting the significant performance enhancements that have been put in place to surpass typical response times found in comparable high-performance systems.

To facilitate autoscaling, the routing layer also emits metrics that inform the Modal Autoscaler for managing container counts based on in-flight request thresholds. This proactive scaling ensures the infrastructure can adapt quickly to fluctuating workloads while minimizing downtime.

Key Features of Modal Servers

Aside from enhanced latency, Modal Servers support several key features conducive to modern application development:

• Proxy Authentication: To enhance security and prevent denial-of-service attacks, Modal incorporates proxy authentication at the routing layer, allowing control over what requests reach user containers.
• Mirroring for Experimental Traffic: Unified handling of traffic allows for A/B testing and active monitoring of inference systems, critical for understanding the effects of changes and gathering data for continual improvement.
• Robust Analytics and Debugging: Tools and systems built into the routing flow provide enhanced visibility into performance metrics and errors, enabling developers to respond quickly to any issues that may arise.

Conclusion: Strengthening Core Infrastructure for Future Workloads

Modal's introduction of Servers presents a vital step in modern cloud computing, allowing users to better manage low-latency applications that require efficient server communication. By focusing on indispensable computational elements and tight integration with emerging technologies, Modal is positioned to cater to the evolving needs of developers and enterprise users aiming to maximize their applications without compromising performance.

The shift emphasizes the need for systematic design in server architecture, ensuring that high-performance applications can thrive in the marketplace of ideas where speed and reliability are paramount. Modal welcomes developers to explore these new capabilities, accessible via their SDK, paving the way for a host of new applications in the growing fields of machine learning and artificial intelligence.

If you're interested in tackling infrastructure challenges and seeking to build strong foundational systems, Modal is actively seeking passionate individuals to join their team.