Modal Enhances Dicts with Unlimited Storage and Smart Caching Features

Modal has upgraded its Dict primitive, providing users with an efficient key-value store suitable for time-sensitive data (TTL). Dicts allow access from any container within the same workspace, making them ideal for tasks like caching function call results and facilitating communication between multiple containers.

The latest improvements introduce new features that enhance the functionality and performance of Dicts, including unlimited storage limits, smarter caching strategies, and a new locking mechanism to ensure data integrity.

Key Changes in Dict Functionality

Here’s a breakdown comparing the legacy version of Dicts with the newly introduced features:

These advancements will apply to all new Dicts created henceforth. The implications of these changes are significant:

• With an unlimited storage capacity, developers no longer need to manage data eviction, allowing for efficient caching strategies where frequently read items can persist as needed.
• The introduction of distributed locking prevents race conditions in environments with high concurrency, ensuring "exactly once" execution semantics using the skip_if_exists functionality.

Optimizing Backend Load with Caching

Let’s consider a typical application architecture currently being employed on Modal where caching can optimize performance. This includes an "expensive" function that has noticeable latency. A basic implementation could resemble the following:

@app.function()
def expensive_function(x: int) -> int:
    time.sleep(30)
    return x ** 2

@app.function(image=modal.Image.debian_slim().pip_install("fastapi[standard]"))
@modal.concurrent(max_inputs=100)
@modal.fastapi_endpoint()
def expensive_function_endpoint(x: int) -> int:
    expensive_function_modal = modal.Function.from_name(APP_NAME, "expensive_function")
    return expensive_function_modal.remote(x)

Post-implementation analysis often reveals that users are submitting the same few requests repeatedly, showing a preference for specific computations.

Previously, developers would encapsulate function calls within classes designed to manage request caching. An example structure could be:

class RequestCacher():
    """Utility class using `modal.Dict` to issue deduped requests and cache results."""

    def __init__(self, function: modal.Function):
        self.function = function.hydrate()
        self.cache = modal.Dict.from_name(f"{function.object_id}-cache", create_if_missing=True)

    def call(self, request_id: bytes, *args, **kwargs) -> Any:
        pass  # Implementation goes here.

Standard logic checks whether an entry is already cached; if not, it proceeds to execute the function and stores the result. However, simultaneous requests can lead to redundant backend calls before the cache is primed.

The introduction of Dicts allows developers to create a more efficient request handling mechanism to minimize these redundant calls.

Implementing Deduplication with Locks

The updated request logic now includes mechanisms for handling duplicate requests efficiently:

• Attempt to acquire a lock by placing a pending marker in the Dict if it isn’t already active.
• If a request is already underway, read from the Dict and fetch the result.
• If the pending entry was created successfully, execute the function call and store its reference as an in_progress entry until completion.

A basic implementation could look like this:

def call(self, request_id: bytes, *args, **kwargs) -> Any:
        pending = _CacheEntry(_RequestState.PENDING, time.time())
        if not self.cache.put(request_id, pending, skip_if_exists=True):
            return self._fetch_cached_result(request_id)

        function_call = self.function.spawn(*args, **kwargs)
        in_progress = _CacheEntry(_RequestState.IN_PROGRESS, function_call)
        self.cache.put(request_id, in_progress)

        result = function_call.get()
        completed = _CacheEntry(_RequestState.COMPLETED, result)
        self.cache.put(request_id, completed)
        return result

Using this strategy, developers significantly improve response times, reduce backend load, and ensure that expensive computations occur only once, enhancing overall user experience.

For those ready to utilize the new Dict features, creating a new instance is a straightforward process—whether for caching, implementing locking strategies, or managing other states. For detailed documentation, visit the docs. Users wanting to leverage the skip_if_exists feature may need to upgrade their client version.

If you have questions or want to share your projects, join our discussion on community Slack.