Substack Enhances AI and ML Workflow Efficiency by Transitioning to Modal

Substack is widely recognized as a leading platform for writers to produce and distribute newsletters, currently boasting over 17,000 writers and generating approximately $300 million in paid subscriptions. This platform has transformed the way independent writers engage with their audiences, allowing them to build direct relationships and monetize their content effectively.

Substack leverages machine learning in multiple applications such as spam filtration, newsletter suggestions, audio transcription, sentiment analysis, and image generation. In a strategic shift, the company has transitioned both the training and deployment of these models from AWS SageMaker to Modal.

The Challenges with AWS SageMaker

Previously, Substack relied on AWS SageMaker and orchestrated its machine learning workflows with Airflow. However, integrating or updating models became an arduous task due to several factors that complicated the engineering workflow.

First, the overall developer experience was frustrating. Engineers had to navigate a complex setup process that involved entering the SageMaker interface, creating a notebook, specifying machine requirements, and waiting for resources to initialize—all before even writing any code. This cumbersome process often led to wasted time, which is particularly detrimental in the fast-paced tech environment. The task was compounded by the challenge of managing multiple remote environments, including Jupyter notebooks, SageMaker training machines, and production infrastructure. If you’re working in this space, you know how critical it is to minimize friction in your workflow.

Second, collaboration was hampered. According to Mike Cohen, Head of AI and ML Engineering at Substack, the process often required teams to replicate code across different notebooks. This lack of code sharing made it difficult to maintain consistency across projects and slowed down development. Without a unified process for disseminating and updating code, teams risked duplicating efforts, leading to inconsistencies and potential bugs in production.

Lastly, the delays in container startup times hindered productivity. Engineers frequently faced waits of over five minutes to get training machines operational. This is more significant than it looks; the constant interruptions disrupt the concentration needed for coding and analysis. Substack's exploration of AWS’s Serverless Inference product, which promised a speedier experience, was underwhelming due to its lack of GPU support and still failed to meet sufficient speed requirements, further aligning with the challenges they faced. In machine learning, speed is often just as crucial as model performance itself.

Accelerated Workflows with Modal

Upon transitioning to Modal, Mike Cohen immediately noticed a more intuitive developmental environment. Unlike SageMaker, Modal allowed developers to focus on coding rather than managing infrastructure. He initially tested Modal with a transcription model and found that it simplified the iteration process significantly. Development and testing were conducted directly in the code editor and command line, eliminating the cumbersome switches between interfaces associated with SageMaker. This shift illustrates a significant cultural change — one that prioritizes developer productivity and satisfaction.

Implementing an inference endpoint in Modal took just about an hour, a stark contrast to the lengthy procedures previously needed that could absorb entire workdays. The implications of this efficiency can't be overstated. Modal's built-in autoscaling capabilities enabled seamless parallelization of transcription workloads right out of the box, which means that expanding a machine learning application now requires far less logistical effort and expertise. Scalability no longer comes at the expense of operating speed.

Recognizing the advantages of Modal, Substack's team decided to migrate their existing models' training and deployment as well. They established a comprehensive workflow encompassing fine-tuning, validation, and deployment by utilizing Modal’s native storage features. This process included several key steps:

1. Training data was fetched from Snowflake, saved to S3, and then mounted to Modal.
2. Model fitting was executed using Modal functions.
3. Model weights were saved to a Modal network volume.
4. Validation of the model was performed with a separate Modal function using the saved weights.
5. Finally, the newly trained model was deployed on Modal, and a Modal key-value store was updated to track the current production version.

With these changes, Substack has achieved a previously unattainable level of speed and flexibility in developing and deploying machine learning workflows, essential for evolving their recommendation systems and other AI-driven applications. The ability to iterate quickly on models means Substack can remain competitive, adapting to users' needs and preferences almost in real-time.

Implications and Future Outlook

This shift from AWS SageMaker to Modal not only represents a technological enhancement but also signals a broader trend in machine learning workflows. As companies seek more streamlined, developer-friendly solutions, the demand for platforms that minimize complexity will only increase. The ability to focus on coding rather than deployment logistics can lead to a richer array of applications, making AI feel more accessible to organizations of all sizes.

For those in the machine learning space, it's important to recognize the potential shifts in strategy that can come from adopting new tools. Substack's experience shows that the right infrastructure can facilitate innovation and speed but also underscores the risks of remaining tied to outdated systems that may inhibit growth.

As machine learning becomes more integrated into applications across various sectors, companies will have to continually reassess their technology stacks. The migration to Modal could very well be a case study for others facing similar challenges. Is your current approach holding you back? That's the question you might want to ask.