Doppel Enhances Machine Learning Workflows with Modal to Streamline Experimentation and Inference

Doppel, an AI-native platform focused on detecting and disrupting social engineering attacks, has recently transitioned a significant portion of its machine learning (ML) workflows to Modal. This shift aims to tackle two primary requirements of their ML operations: the need for rapid experimentation and reliable, scalable inference.

Challenges in ML Workflows

Before adopting Modal, Doppel faced considerable friction in their ML infrastructure. Their experimentation process was impeded by sequential training runs, which extended the waiting time between testing new hypotheses. Each training job could take hours, causing delays whenever a failure occurred late in the process. Consequently, this inefficiency not only extended feedback loops but also led to bundling multiple changes into single experiments, complicating performance analysis and testing processes.

Transitioning to Modal

By migrating to Modal, Doppel has dramatically accelerated its experimentation capabilities. Modal empowers users to run experiments in parallel effortlessly, eliminating the need for complicated orchestration workflows. Instead of managing job queues or distributed training pipelines, developers can utilize straightforward Python code to parallelize tasks using constructs like map().

One clear application of this parallelization is in K-fold cross-validation. Each fold can run concurrently, rather than sequentially, allowing for quicker evaluation of model performance. For example, defining a fold-level training function and mapping it across all folds enables efficient parallel execution with minimal overhead.

Improving Feedback Loops

Implementing a model that promotes faster iteration cycles has been crucial for Doppel's approach. Their training workflow now consists of a structured loop that includes proposing changes, running experiments, summarizing results, and determining the next changes based on evidence. This cycle has been enhanced by integrating coding agents, which streamline mechanical aspects such as launching experiments and collecting metrics, allowing machine learning engineers (MLEs) to focus on high-level decision-making regarding experimental ideas.

This situation parallels efforts in fully autonomous ML experimentation, such as Andrej Karpathy’s autoresearch project. However, Doppel's approach ensures that human oversight remains central, with agents handling routine steps while MLEs maintain creative control over the experimentation process.

Scalable Inference Infrastructure

In addition to training, the inference process is critical to Doppel's real-time detection capabilities. Models must operate with low latency to respond swiftly to threats and maintain elastic scalability to accommodate unpredictable traffic spikes during attack campaigns.

Previous GCP Inference Setup

Doppel initially deployed its inference models within Google Cloud Platform (GCP). Their previous workflow entailed packaging models into custom Docker containers, setting up Cloud Run services with GPU attachments, and exposing endpoints via HTTP for predictions. While this integration worked reasonably well initially, it created several challenges as the workload expanded.

Deploying models often took considerable time, with building large model images requiring up to 30 minutes, which slowed down iteration and deployment. Additionally, Cloud Run instances could only attach a single GPU, complicating scaling during higher demand. The combination of these operational frictions ultimately led to delays in model deployment and oversight with complex HTTP service layers for authentication and request handling.

Modal's Simplifications

Switching to Modal has streamlined Doppel's inference deployment considerably. Now, image caching and persistent volumes reduce build times by as much as 10x, transforming even complex setups to ready in under a minute, enhancing the pace at which the team can iterate on deployment configurations.

A standout feature of Modal is its serverless architecture, which automatically scales inference workloads according to demand, ensuring that traffic spikes can be managed without manual intervention. This accommodates sudden increases in requests typically seen during phishing campaigns or similar attacks.

Moreover, Modal simplifies the operational overhead by abstracting away the need for an HTTP service layer. Instead of requiring Flask endpoints, inference functions can be invoked directly through Modal’s interface, allowing for a more efficient and less code-heavy deployment process.

Conclusion

The move to Modal represents a pivotal change for Doppel, dramatically reducing the operational costs associated with their machine learning deployment. By overcoming traditional infrastructure challenges, Modal enables Doppel to focus less on operational issues and more on experimentation and innovation in their detection models. As a result, the current bottleneck lies not in infrastructure limitations but in the speed at which new ideas can be proposed and evaluated.

This strategic alignment with Modal enhances Doppel's ability to scale while maintaining agility—a necessity in the ever-changing landscape of cybersecurity threats.