Prompt Engineering Competition: Unearthing Human Skill through the Prompt Olympics

In the evolving field of natural language processing, the art of prompt engineering has garnered attention as a vital skill. The team at Basis has delved into automatic prompt optimization tools, observing promising advancements as these tools often outperform hand-crafted prompts. Despite this success, a lingering question remains: what is the upper limit of human potential in this domain? To investigate this query, Basis initiated the Prompt Olympics, an event that merges competitive programming culture with the discipline of prompt engineering, complete with a $5,000 prize for the winners.

To help facilitate this event, Modal stepped in to provide the necessary infrastructure, allowing participants to focus on refining their prompting skills. This competition featured various challenges that tested participants' abilities in engaging with language models effectively. Below, we detail two key components of the competition: the coding challenge and the finale game.

Challenge: Writing a Prompt for Code Generation

One of the primary challenges involved a task familiar to many developers: participants were required to write prompts that would instruct a language model to generate code capable of passing specific tests. Instead of coding directly, competitors had to carefully craft prompts that elicited the desired output from the model.

The underlying large language model (LLM) for this challenge was LLaMA 3 70B, which had been deployed using Modal’s infrastructure, known for its ease of use and serverless GPU acceleration. This deployment approach streamlined the process for participants looking to utilize their own LLMs effectively; a comprehensive guide was available for those interested in replicating this setup for their purposes.

However, executing and grading the submissions posed unique challenges. Since the generated code could be potentially harmful or malfunctioning—think erroneous commands like sudo rm -rf / or hazardous zip bombs—clicking “execute” wasn't straightforward in this competitive environment. Modal's solution was to employ the Modal Sandbox, a secure and dynamically allocated environment designed specifically for executing untrusted code. This system is equipped with robust features for managing secrets and other resources while also ensuring safe, performant execution.

Finale: Playing Crafter

The concluding aspect of the competition revolved around a gaming challenge in which participants crafted prompts to control a character in the game Crafter. This game, a simplified version of Minecraft, is specifically structured for agent research and requires strategic thinking as players manipulate game mechanics through their prompts without direct character control. The winning entry demonstrated an agent that effectively created a pickaxe, mined coal, and navigated enemy encounters, showcasing the intricacies of prompt crafting in a dynamic environment.

This challenge illustrated another side of programming competitions that extends beyond traditional metrics like latency or test case accuracy. It highlighted how prompt engineers might influence autonomous agents in virtual scenarios, an important aspect for future applications of LLMs.

Nonetheless, Basis’s team recognized that even sophisticated models like LLaMA 3 70B posed significant barriers to effective prompting in complex game scenarios. To overcome these limitations, they opted for a proprietary model API service for the LLM part of the challenge, allowing for more reliable interactions.

Despite this switch, Basis continued to leverage Modal for the overall competition framework. Modal’s ability to provide a general-purpose computing infrastructure combined with serverless capabilities proved invaluable. In addition to hosting the game, the platform facilitated the integration of various services required for smooth operations throughout the event.

The Prompt Olympics not only fostered a competitive spirit but also expanded the understanding of what can be achieved through precise prompt engineering. Participants came away with insights into optimizing model interactions, which could impact various applications in software development and beyond.

For those interested in the specific outcomes of the competition, including the top-performing prompts, a detailed recap is available on the Basis blog.

Modal's collaboration with Basis in this initiative reflects a shared appreciation for the challenges of programming competitions. Many among the Modal team have experienced firsthand the engaging learning journey that competitive programming offers. If you share this passion, know that Modal is continuously looking to expand its team!