sync. Transforms Video Lipsyncing with Modal to Deliver 100+ Hours of Video Daily

Overview of sync.

sync. is a research laboratory specializing in the creation of foundational models designed to understand and manipulate human interactions within video content. Their state-of-the-art lipsyncing model allows for the reanimation of speech in various video formats—be it live-action, animated, or AI-generated. The team is renowned for their previous work on wav2lip and currently serves thousands of developers and creative professionals globally.

One impressive demonstration of their technology showcases a zero-shot lipsyncing model that maintains the unique speaking style of actors, like Nicolas Cage, even when translating into different languages. This application signals a noteworthy advance in maintaining the integrity of the original performance through AI.

Their journey to success began with a viral demo featuring a conversation between Lex Fridman and Mark Zuckerberg discussing Elon Musk in Hindi, a moment that propelled them into the AI video scene.

The Initial Challenges: Outgrowing Google Colab

Initially, the team's viral success relied on a Google Colab notebook. While effective for rapid prototyping and research, it fell short as a sustainable solution for scaling a business. Despite its origins, they quickly ran into deployment challenges when trying to transition to AWS Lambda. Issues such as complicated deployment processes, restrictive autoscaling policies, and insufficient GPU support hindered their progress.

At this point, the sync. team consisted of just five members. Their primary objective was to advance their model training and product iterations, rather than managing complex MLops infrastructures. Various AI-as-an-API providers were explored, including Replicate. Although effective for standard model deployment, they found it difficult to adapt their custom models using Replicate’s Cog framework, where every code change necessitated rebuilding and redeploying containers—taking between 10-15 minutes each time. This delay was impractical for a startup focusing on rapid iteration.

Adopting Modal: A Seamless Fit for Video Editing

Through connections with Founders Inc and Y Combinator, sync. discovered Modal’s startup credits initiative and successfully secured $25,000 in credits. Modal’s service distinguished itself from others, providing a straightforward process without requiring the team to log in to separate platforms. Instead, they could enhance their existing code, initiate modal serve for real-time updates, and confidently deploy, knowing that Modal would effectively manage autoscaling.

This ability to iterate quickly allowed sync. to deploy updates up to 95 times a day, testing multiple small iterations while effectively managing a live production workload. Over the course of a year, they launched 10 significant model variants and conducted about 1000 iterations in the interim. This rapid iteration process is essential for staying competitive in a dynamic development environment.

Deep Dive: Processing Over 100 Hours of Video Daily

To illustrate the efficiency of Modal in practice, consider sync.'s approach to lipsyncing over 100 hours of video daily. Their processing strategy organizes videos based on duration. Shorter clips are handled as a single batch, while longer videos undergo a systematic breakdown:

1. The video is divided into n scenes.
2. Face detection and translation tasks are executed across n T4 GPU containers.
3. The proprietary LipSync GenAI model processes each scene on A100 GPUs, which are more powerful.
4. Finally, the dubbed scenes are reassembled into a complete video.

This process exemplifies how Modal’s parallel execution model enables efficient handling of compute-intensive workloads that would otherwise require extensive time and resources. The streamlined processing allows for significant scalability without sacrificing quality or speed.

Future Prospects for sync. and Modal

Looking ahead, sync. aims to democratize access to advanced video effects capabilities, essentially putting the tools of a high-end VFX studio in the hands of everyday users. While their current focus is on translation and lipsyncing, they are also actively developing features related to emotion editing, pose adjustments, and even modifications to physical characteristics of video subjects.

The accomplishments of sync. highlight the potential of modern startups in the AI realm. By leveraging Modal's infrastructure, they can direct their efforts entirely on R&D, enabling the rapid delivery of products that push the boundaries of AI technology in video editing.