Open ASR Models Enable High-Performance Speech Transcription at Lower Costs

Emerging Open ASR Models

With the success of generative modeling demonstrated by ChatGPT, the landscape of artificial intelligence services has predominantly featured proprietary APIs. However, a notable shift is occurring: open weight models are gaining momentum across various domains, including natural language processing, image recognition, and video analysis. In recent months, there's been a surge of efficient open-source models specifically targeting automated speech recognition (ASR), commonly referred to as speech-to-text (STT) technology.

This new wave includes impressive releases like NVIDIA's Parakeet and Canary, as well as Kyutai’s STT. These models showcase remarkable metrics such as low word-error rates (WER), high real-time factors (RTFx), and value-added features like multilingual processing and vocal activity detection (VAD).

Cost and Performance Analysis

Conducting a simplified cost analysis reveals intriguing insights about these new ASR models. Proprietary APIs typically charge around 0.4¢ per audio minute. Meanwhile, the open ASR models demonstrate RTFx rates that allow thousands of audio minutes to be processed per wall clock minute

On platforms like Modal, running an A100 or L40S GPU incurs approximately 4¢ per minute. This suggests that even with conservative estimates concerning overhead, organizations should be able to transcription costs significantly less than proprietary services.

Curious about these claims, we implemented a batch transcription service on Modal using NVIDIA’s Parakeet and Canary models. We compared this to a popular proprietary API over a week’s worth of speech data sourced from the HuggingFace ASR Leaderboard.

Our results validated earlier assertions: the implementation yielded over 100x improvements in either speed or lower costs when compared to the proprietary service, while demonstrating a comparable error rate, and in some instances, a slightly better performance, achieving:

• One week of audio transcribed in just one minute.
• Total cost of just $1.

Implementing Efficient Transcription Processes

To achieve our transcription goals, it’s crucial to understand the metrics guiding our optimizations. Our primary objectives included enhancing both speed and cost-efficiency. The ideal scenario for large-scale batch transcription typically arises in environments where recordings accumulate rapidly, such as call centers that generate thousands of hours of audio continually. Similarly, organizations accumulating audio for model training benefit from fast and cheap transcription services.

In these settings, the batch transcription approach diverges significantly from streaming scenarios. The two modalities possess distinct metrics and implementation requirements, often necessitating different strategies and designs.

Benchmarking Strategies in Batch Transcription

The performance of ASR systems, particularly in terms of accuracy, can be evaluated using WER and benchmarks such as the ESB and HuggingFace ASR Leaderboard. Utilizing these frameworks allows for consistent validation and reassurance that deployments do not degrade performance. While models often report speed based solely on RTFx, our focus extends to end-to-end performance—essential in cloud applications.

In distributed transcription, one must also consider factors such as cold start times and data transfer durations. The paramount concern lies in the total time required to process jobs rather than the execution time of the models alone. This comprehensive perspective informs our throughput reporting which encompasses all processing durations.

Constructing Batch Transcription Workflows on Modal

Leveraging Modal, we established batch transcription workflows for both open models and proprietary APIs, ensuring we tuned for maximum performance. The architecture diverged: for open models, transcription occurred directly on Modal's GPUs, while the proprietary API utilized CPU containers for HTTP requests, omitting CPU costs from our efficiency calculations.

Deploying NVIDIA’s Models on Modal GPUs

NVIDIA's ASR models are among the fastest and most precise available. We particularly focused on the Parakeet model, which is known for high-performance and low error rates, along with Canary, designed for multilingual tasks.

Deployment involved utilizing NVIDIA’s NeMo framework which simplifies switching between different ASR models. By adapting code available from the HuggingFace Open ASR GitHub repository, we quickly established a distributed batch service that automatically provisions GPUs for each task.

Maximizing Throughput with a Proprietary API

Testing a competitive proprietary ASR API required us to optimize the deployment to maximize throughput. Here, we distributed the transcription requests across multiple workers, aligning the quantity with the API’s maximum concurrent limits to achieve fair performance comparisons.

Despite experiencing throttling, we focused on reporting the maximum throughput achieved, ensuring a reliable basis for comparison. While our measurements may not align precisely with the documented performance metrics, they still provide a close representation.

Engineering for Exceptional Throughput

Effective data distribution is foundational for maximizing throughput in any computational task. Distributing tasks poorly can lead to throughput reductions instead of gains. Therefore, thoughtful system architecture and continuous experimentation are critical for success.

Optimizing Batching Techniques

One of the primary considerations involved efficiently packing audio data for transcription, balancing both memory and time efficiency. Proper optimization helps eliminate unnecessary iterations or requests, delaying processing.

To manage batches effectively, we minimally modified how audio files were grouped before submission to ensure even workloads across all Modal workers. Essential factors included byte count, audio file quantity, and total audio duration—all crucial in achieving balanced workloads.

Additionally, many ASR models, including Parakeet, support simple batch processing commands, enhancing GPU activity within our applications. The utilization of appropriate batch sizes based on NVIDIA's recommendations ensured effective use of GPU memory.

Transferring Data Efficiently

Data movement constitutes another key element influencing performance. In our setup, audio segments were initially hosted as WAV files on Modal, with parallel downloads optimizing network utilization. Applying multithreading to the download process minimized latency and helped ensure efficient data transfer to workers.

Determining Optimal GPU Configuration

Choosing the right type of GPU and corresponding request numbers proved essential for optimizing cost-effect ratios. By empirically testing configurations, we identified a range of optimal setups demonstrating varied performance metrics. The proprietary API’s performance would likely reside beneath the prominent performance thresholds established by our open models.

Deploying the leading open ASR models on Modal provides a competitive edge over proprietary solutions, allowing organizations to extract value from transcription processes more efficiently than before.

Launching Your High-Performance Transcription Service

Organizations like Substack and Zencastr have successfully harnessed Modal for scalability in their transcription workloads, eschewing the need for proprietary APIs or specialized engineering certifications. Users can establish batch transcription services with just a few lines of Python code.

Explore how our platform can enhance your audio inference tasks, set up batch transcription effortlessly using the code provided, or browse our additional speech-to-text examples for further insights.