Optimizing Filesystems for Virtual Machines: Modal's ImageFS-blk Solution

At Modal, performance is paramount as we cater to extensive GPU workloads and a multitude of agent sessions demanding rapid execution. Our commitment to optimizing system efficiency is reflected in our latest effort: enhancing the filesystem stack for our VM Sandboxes, which needs to combine exceptional speed with sophisticated functionalities.

Recently, we introduced VM Sandboxes to support increasingly complex workloads, all while maintaining the speed expectations set by our gVisor-based platform. Central to the performance of these Sandboxes is the filesystem, which acts as the backbone of their operation. Today, we explore how we evolved our filesystem capabilities to optimize VM performance.

Background: ImageFS and Rapid Cold Starts

In April 2022, we tackled the “container filesystem” challenge by launching ImageFS, a distributed filesystem designed for lazy-loading containers. ImageFS operates by fetching only the minimum required metadata upfront, utilizing a Rust-based FUSE (Filesystem in Userspace) server for mounting. The filesystem fetches content from a tiered cache, including local memory and regional content delivery networks (CDNs), all the way up to global blob storage. With smart prefetching in place, accesses typically hit files already in local disk or memory.

ImageFS is a significant contributor to Modal containers achieving cold starts in under a second. Unlike traditional Docker setups, which can take minutes to pull layers, ImageFS can quickly load its metadata index in milliseconds. Even with its FUSE architecture, ImageFS delivers performance similar to non-FUSE filesystems by signaling the kernel to aggressively cache responses to requests.

For write operations, we implemented OverlayFS, which merges a read-only layer (ImageFS) with a writable upper layer, facilitating file mutations.

Leveraging ImageFS for Virtual Machines

As we designed our VM solution, we aimed to build upon ImageFS to emulate the high-performance sandbox experience known from containers. Containers share the host system’s Linux kernel and therefore can directly utilize ImageFS. VMs, conversely, use an independent guest kernel, which cannot access the host's filesystems, including ImageFS.

To bridge this gap in the VM ecosystem, we employed virtio-fs, another FUSE-based filesystem that transmits guest VM filesystem operations to a virtio-fs daemon (virtiofsd) on the host. This daemon interprets and applies filesystem requests safely, allowing VMs to use ImageFS and inherit its low-latency characteristics.

Upon implementation, VMs booted using the same modal.Image as containers, achieving desired cold-start performance. However, our initial assumptions overlooked critical performance metrics.

Overcoming Slowness: Performance Challenges with virtio-fs

While quick cold starts are valuable, they are only one aspect of filesystem performance. By layering virtio-fs on top of ImageFS, we inadvertently introduced latency in filesystem operations. For instance, a typical recursive chown command took over a minute; standard installations via pnpm and bun became three times slower, and random file searches via ripgrep suffered significant delays.

To diagnose the issue, we dissected the system calls made during our benchmarks, comparing ImageFS performance with and without virtio-fs. A crucial finding revealed that operations like stat were twice as slow under virtio-fs, and chown experienced a six-fold increase in latency. Further analysis uncovered that each chown command was decomposed into multiple operations requiring round trips between the guest VM and the host.

1. Initial chown call leaves guest as fchownat.
2. Metadata check requires host syscalls for attributes and capabilities.
3. Ownership change necessitates another round trip for file attributes.

In total, this single filesystem operation resulted in three round trips from the VM to the host and six syscalls—each incurring additional overhead.

This structure highlighted a fundamental limitation in virtio-fs, as it functions effectively as a shared filesystem, which incurs asynchronous communication overhead even when isolation from the host is not necessary. Faced with these inefficiencies, we reevaluated our approach.

Switching Paradigms: Introducing Block Devices

By allowing the filesystem to reside entirely within the VM, we theoretically could eliminate the need for cross-VM communication, requiring only the host to maintain the raw data blocks. Essentially, this approach utilizes block devices, which operate independently of any file structure. Block devices treat data as unformatted storage, optimizing communication between the host and VM given their established efficiencies.

The Linux kernel's proficiency with block devices extends back decades, making it a natural fit for both physical and virtual machines utilizing virtio-blk, converting all operations to reside within the guest. This drastically changes our implementation of filesystem operations, removing the need for metadata requests from the host to the guest kernel.

ImageFS-blk: Merging Speed with Lazy Loading

To reconcile the benefits of block devices with the lazy-loading properties of ImageFS, we developed ImageFS-blk. Rather than relying on mkfs.ext4, which necessitates all data to be pre-loaded, ImageFS-blk facilitates on-demand data provisioning.

By implementing a hybrid strategy, we mimic ext4's metadata functionalities while serving file content as required. The block device interface allows the guest kernel to maintain its capability to manipulate filesystem metadata locally, which improves performance significantly due to the reductions in inter-process communication.

When the guest writes to the filesystem, the ImageFS-blk service marks updates in a bitmap, preserving writes exclusively from the VM while efficiently managing snapshot operations. The read process similarly maps block requests back to file data, utilizing existing metadata to link retrieval operations to the actual content in ImageFS.

Enhanced Performance Benchmarks

In terms of performance, ImageFS-blk vastly outstripped the virtio-fs implementation. For example, our benchmarks showed that bun install times decreased from 12.5 seconds to 1.07 seconds, even outperforming ImageFS due to the elimination of outside calls for metadata operations. Performance improvements extended to database applications, where metrics from workloads like pgbench illustrated marked efficiency gains.

Our team remains dedicated to refining filesystem performance, aiming for further optimizations and efficiency improvements.

Final Thoughts

For developers eager to experience the advancements we've implemented, our VM Sandboxes are now generally available with the new ImageFS-blk backend. We invite those interested in this field to explore our documentation and consider joining our team to push the boundaries of filesystem performance further.

Acknowledgements

Thanks to our dedicated team members—Daniel Norberg, Staffan Gimåker, Apoorva Bhagwat, Connor Adams, and Jonathon Belotti—for their invaluable contributions to this project.