Modal Auto Endpoints: Empowering Teams to Own Inference Infrastructure

Modal is introducing Auto Endpoints, a streamlined solution designed for teams like Cognition, Decagon, Fathom, and DoorDash. These endpoints allow users to take ownership of their inference without sacrificing cost efficiency or developer speed. With a simple command, teams can create and deploy their models, making the transition to production-grade inference straightforward.

To illustrate, a team can initiate an endpoint with a single command:

modal endpoint create --name agent --model zai-org/GLM-5.2-FP8

Ownership in Inference

When using proprietary models from third-party providers, organizations often face limitations. They may find that model performance can decline unexpectedly or that access can suddenly be revoked. In essence, lacking control means lacking autonomy over the inference process. Modal posits that true ownership extends beyond merely using an API; it involves comprehending and optimizing the underlying code executing the inference.

Most managed inference services present a black box, limiting what users can observe and modify. Prior to Auto Endpoints, gaining full control meant developing and maintaining a bespoke inference service, resulting in assuming responsibility for aspects such as engine tuning, container management, and autoscaling. Recognizing this challenge, Modal designed their Auto Endpoints to offer many of these capabilities while streamlining the user experience.

Auto Endpoints stand apart from traditional inference solutions by providing an OpenAI API-compatible service that maintains transparency and user control. This structure allows users to see and modify components that drive inference, from GPU configurations to engine flags. The service supports various models, notably the GLM-5.2, and principal differences include the following attributes:

• No Hidden Code: Users gain visibility into the code that drives inference, from GPU selection to inference engine settings.
• No Obscured Metrics: Performance metrics necessary for debugging, such as latency and queueing details, are readily available on dashboards.
• No Sales Requirement: Users can deploy models without needing to engage in lengthy sales processes, deploying models through a CLI command instead.

Infrastructure Designed for Inference

Modal's architecture is grounded in a robust infrastructure that supports low-latency inference. This foundation allows teams to scale their models dynamically, aligning resource allocation with actual demand rather than reserved capacities. The platform provides a high-performance autoscaling feature and a custom container runtime designed to handle varying loads efficiently.

Notably, Modal has integrated a new component, Modal Servers, which ensures ultra-low-latency routing, allowing HTTP requests to be processed with just five milliseconds of overhead. Modal Servers ensure that scaling is seamless, removing the need for queueing while sustaining reliability. Such a framework ensures that as demand rises, additional replicas can be deployed automatically, maintaining performance without manual intervention.

Streamlined Performance Management

When it comes to inference engines, achieving optimal performance can be comparable to managing database systems; they require careful tuning to extract the best results from the hardware. Traditionally, teams might have defaulted to proprietary solutions, relinquishing control over performance tuning to external providers.

Auto Endpoints simplify this process by offering deployment configurations informed by industry best practices. Users are not required to specify GPU types or adjust complex settings until they are ready for those advanced customizations. This approach enables teams to focus on their key workflows rather than the intricacies of performance tuning.

Furthermore, Modal is dedicated to enhancing performance through open-source collaboration. By iterating upon established engines and refining model performance via techniques such as speculative decoding, Modal ensures that users consistently receive upgrades that enhance their endpoint efficiency.

Observability and Metrics

Modal promotes not only performance but observability, allowing users to gain insights into their inference operations. Equipped with server and inference metrics, users can access a variety of performance indicators, creating a well-rounded view of system functioning.

• Server Metrics: Include temperature, power consumption, and utilization data for GPUs.
• Inference Metrics: Provide standard indicators such as time-to-first-token and inter-token latency.

These metrics exceed the standard offering from many inference service providers, providing a comprehensive perspective on performance that allows teams to identify bottlenecks and troubleshoot effectively.

Automation and Continuous Improvement

The design of Modal Auto Endpoints emphasizes a forward-thinking approach—wherein the engineering of inference services becomes increasingly automated. The platform's interfaces are designed to facilitate the application of best practices for configuring and optimizing deployments, leading teams toward enhanced operational efficiency.

Modal aims to automate various processes, from benchmarking configurations to retraining models based on production performance. The goal is to reduce manual intervention, enabling continuous improvement of inference outcomes through data-driven insights and automated adjustments.

Conclusion and Next Steps

Modal's Auto Endpoints offer a compelling solution for teams looking to take complete ownership over their inference environments. By providing transparency, control, and automation, teams can ensure they are well-positioned for both current needs and future growth. Interested users can sign up and start deploying Auto Endpoints here.