Enhancing QR Code Generation with Evals and Inference-Time Compute Scaling

In the rapidly evolving field of generative AI, two techniques—evals and inference-time compute scaling—have taken center stage. These approaches are essential not just for the latest trends, such as RAG chatbots or automated coding systems, but serve as the backbone for applications using foundation generative models. This article details our efforts to implement these techniques to enhance the quality of QArt codes, which embed QR data within aesthetically pleasing images.

Generating Aesthetic QR Codes

QR codes have become ubiquitous for sharing information quickly, but their typical design lacks visual appeal. Our goal was to create beautiful QR codes that retained their functionality. We started with QArt codes, a concept inspired by a post from Reddit user nhciao on June 5, 2023, who demonstrated the possibility of using ControlNet to steer diffusion models in crafting QR codes.

ControlNet allows for manipulating images based on an initial pattern by controlling aspects like pose and size—a powerful strategy that enhances creativity beyond simple prompts.

QR codes function by encoding data through patterns of contrasting colors that must be recognized by scanning devices. These codes incorporate sophisticated error correction features to ensure readability even when partially damaged. This ability means that modifications to the typical QR code pattern can be accommodated without losing functionality, as long as the encoded message remains intact.

When leveraging ControlNet to generate QArt codes, the potential arose to creatively distort QR patterns without compromising their scannability. This drove multiple teams, including ours, to participate in the trend, eager to explore its creative opportunities and commercial implementations.

Challenges of Scannability

However, early implementations often faced the principal issue of the QR codes not being readable. Just producing aesthetically pleasing images doesn’t equate to creating functional QR codes. For instance, we produced some images that resembled QR codes but ultimately failed to be scanned due to inadequate structural elements, like the finder pattern.

Not quite a QR code!

In addressing this concern, we prioritized producing scannable QR codes. Initial results aligned well when prompting with simple textures rather than complex artistic scenes, allowing us to use QArt codes in various applications without major hassle. This approach enhanced our confidence in creating visually engaging QR codes that met functional requirements.

A QArt code in the slides for a talk by one of the authors.

The results reflected an emerging opportunity for creativity and utility in generating QR codes, and we expanded our use of QArt codes into places previously dominated by standard QR designs.

Engineering Quality with Evals

The subsequent challenge was to systematically evaluate and improve the quality of these QArt codes. Traditional software evaluation metrics often don't apply to machine learning systems effectively. Instead, neural networks resemble analog systems with inherent variability, requiring tailored evaluation criteria.

To proceed, we established two primary goals for our solution:

1. Ensure images are scannable by standard devices.
2. Maintain a level of aesthetic appeal.

Given the subjective nature of aesthetics, we focused primarily on scan rates for our evaluations. By explicitly defining operational criteria—such as whether an image could be scanned by a smartphone—we set the foundation for our evals.

To automate scanning evaluations, we chose the QReader library, which utilizes a YOLO model to detect QR codes through various transformations. We also employed a machine learning model from the Stable Diffusion community to assist in predicting human aesthetic ratings.

While automation was critical, we needed to ensure that our automated processes aligned with our original human evaluations. Iterative human checks were essential to confirm the reliability of the tools, and we scaled our evaluations to consider thousands of prompt-URL combinations.

You may not like it, but this is what peak ML engineering performance looks like.

This evaluation strategy, though straightforward, allowed us to rigorously assess the outputs and maintain quality while minimizing unnecessary complexity. We also utilized Weights & Biases' Weave for experiment management and data visualization, enabling the team to collaboratively analyze results efficiently.

An example W&B Weave chart grouping generations by version and scan rate.

The importance of a robust eval system cannot be overshadowed; it’s a cornerstone for responsible machine learning engineering. This approach allowed us to gather necessary insights that informed further developments.

Scaling Inference-Time Compute

With operational evals in place, our final goal revolved around achieving a 95% scan rate without sacrificing aesthetic quality on a large scale. The manual processes we employed initially provided a solid understanding of system capabilities but could not serve long-term efficiency. Thus, we transitioned to computing on a larger scale.

Using our evals effectively enabled us to move beyond manual checks to automatic evaluations. The efficiency of reevaluating numerous configurations helped us optimize output while ensuring quality remained consistent. For instance, adjusting key parameters in our generative workflows enabled us to experiment extensively without incurring monumental compute costs.

Our goal evolved into producing multiple QR codes per prompt while ranking the most effective and aesthetically pleasing outputs. By generating eight variations at once, we achieved the targeted scan rate while also enhancing overall quality.

The outcome of our methods not only matched but often exceeded our prior quality benchmarks. Collaborating closely with evaluation processes and inference scaling accelerated our development timeline significantly.

Looking Ahead

Our journey reflects the evolving applications of generative models and how effective engineering strategies can amplify their capabilities. As we foresee the technology maturing, the emphasis will increasingly focus on resolving complex challenges through automated evaluative frameworks.

Building efficient tools that can deal with uncertainties, parameter adjustments, and intricate quality metrics will define how organizations leverage generative AI moving forward. We invite others to explore similar paths and consider tools like Modal, which provides scalable, serverless platforms that simplify practical implementations of generative models.

You can experience our work firsthand by visiting QArt.codes. We wholeheartedly believe in the potential of generative AI, and our work serves as an initial model for future research and applications.

We appreciate the contributions from the community, specifically thanking u/nhciao for the initial inspiration and collaboration with others who developed useful resources for QArt code generation.

With a firm grounding in objectives, methods, and tools, the opportunities are rich and transformative in the realm of generative models. We’re excited about the continued evolution of the technology and its impact across various sectors.