How Zencastr Achieved Massive Audio Transcription Scalability with Modal’s Infrastructure

Zencastr has established itself as a prominent platform for podcast creators, offering comprehensive tools for recording, producing, and hosting shows. With thousands of users generating millions of episodes, the need for high-quality production tools, including AI-powered features, has never been greater.

Among its notable offerings is an advanced transcription service that includes features like forced alignment, verbatim transcription, and timestamps. Initially relying on traditional CPU-based transcription models, Zencastr made a strategic shift toward modern transformer-based models such as Whisper. This transition not only improved transcription quality but also leveraged GPU capabilities for efficient processing.

Challenges with Previous Infrastructure

As Zencastr expanded, the complexities of its infrastructure began to surface, prompting a reevaluation of its existing GPU management strategies. Initially, the team utilized Kubernetes to oversee GPU nodes, but this approach came with significant drawbacks.

• High Costs and Over-Provisioning: Zencastr's transcription demands exhibit a pattern of peak usage during specific hours, leading to a situation where GPU nodes had to remain active around the clock. This resulted in substantial over-provisioning, inflating costs without corresponding benefits.
• Operational Overhead: The need to support various machine learning models created a chaotic scenario for the development team. With each model requiring specific GPU driver versions and software dependencies, maintaining synchronization became a time-consuming and error-prone task.

According to Alex Cannan, Zencastr’s ML Team Lead, managing these GPU resources in Kubernetes was not only expensive but also difficult to maintain. "We had a Kubernetes cluster with some GPU nodes, but maintaining a GPU node all the time in there would be expensive. Some models used the latest Torch version while others relied on older TensorFlow. Keeping everything in sync was brittle and time-consuming," Cannan noted.

Zencastr explored other AI platforms, but found them lacking in crucial areas such as efficient cold starts for custom models and flexibility in scaling operations—two aspects critical to their transcription needs.

Transitioning to Modal for Better Scalability

Upon experimenting with Modal, Zencastr discovered an optimal solution that addressed both cost-efficiency and performance speed. Modal's platform offered several key advantages:

• Dynamic Cost Management: Modal enabled Zencastr to benefit from a scale-to-zero model, meaning the company only incurs expenses while jobs are actively running, eliminating idle GPU costs.
• Streamlined Environment Management: The introduction of Modal Images, which allow for programmatic environment specification, vastly simplified dealing with model dependencies. Automated updates to CUDA drivers for GPU containers further alleviated infrastructure management complexities.
• Experimental Flexibility: Zencastr was able to optimize large jobs, experimenting with different GPU configurations without altering source code. This helped the team assess performance and cost trade-offs effectively.

As a result of this migration, Zencastr has significantly accelerated its development processes. Tasks formerly requiring days—such as environment setup and driver troubleshooting—have been condensed to mere hours, thereby enhancing iteration cycles for the ML team.

Cannan emphasized the effectiveness of the Modal infrastructure: "Modal has been a really nice, scalable solution for us. We don’t have to worry about pre-allocating GPUs weeks ahead of time – we just spin it up and it works."

Currently, all of Zencastr’s core machine learning workflows operate on Modal, which includes:

• AI-Driven Transcription: This is the primary service for transcribing each podcast episode hosted on the platform.
• Speaker Analysis: Features like speaker counting and laughter detection facilitate enhanced content analysis.
• Quality Enhancements: Audio quality scoring and post-production improvements are also powered by machine learning algorithms.

Scaling Batch Audio Processing Efficiently

In its quest to enrich historical audio data, Zencastr frequently runs large-scale batch jobs, which require significant parallel processing. These jobs, essential for the training of new models, involve extracting multiple audio features concurrently.

Using Modal, Zencastr efficiently handles these batch processing tasks by employing a straightforward setup:

1. A Modal Function is deployed for downloading audio files from S3 storage.
2. Files are transferred to a Modal Volume, which allows for rapid access across distributed storage landscapes.
3. Feature extraction is parallelized among different Modal Functions drawing from the same Volume, maximizing efficiency.

During one notable instance, Zencastr successfully scaled to 1,500 concurrent GPUs within minutes to enrich its entire podcast archive. This feat showcased Modal’s resilience and ability to handle substantial workloads without additional operational burden on Zencastr’s team.

Future Directions in AI and Podcasting

Looking ahead, Zencastr is eager to explore new avenues through its collaboration with Modal, particularly in developing voice-adjacent models.

Cannan reflects on the partnership, stating, “Whether it’s peak hours or the middle of the night, Modal gives us the confidence that we're minimizing latency and cost at the same time. Their on-demand GPUs have powered us through centuries of audio with a seamless developer experience and dependable performance.”

With Modal’s support, Zencastr is poised to advance its infrastructure to meet the evolving needs of AI-enhanced podcasting, ultimately transforming how content is created and consumed.