Can Games Like Pictionary and Minecraft Effectively Test AI Ingenuity?

Traditional AI benchmarks, which often rely on static questions or rote learning, fall short of measuring real-world problem-solving abilities. That’s why AI developers are turning to games like Pictionary and Minecraft to test models’ ability to think beyond pre-learned responses and adapt to dynamic situations.

Freelance AI developer Paul Calcraft recently created an app where two AI models play a Pictionary-style game, with one model drawing and the other guessing. Calcraft explained, “I thought this sounded super fun and potentially interesting from a model capabilities point of view.” Inspired by similar projects, he sought to create a benchmark that would challenge AI models beyond mere memorization.

The Logic Behind Games as Benchmarks

Games like Pictionary and Minecraft add a layer of complexity not found in standard benchmarks, Calcraft argues. “The idea is to have a benchmark that’s un-gameable,” he said, as these games can’t be solved simply by recalling specific data points from training. Pictionary, for instance, requires models to understand shapes, colors, and spatial relationships, capturing elements of visual and linguistic reasoning.

Similarly, 16-year-old Adonis Singh created a benchmarking tool called mc-bench, where AI models control a Minecraft character to complete construction tasks. According to Singh, Minecraft’s open-ended nature provides a rich environment for testing resourcefulness. “It’s not nearly as restricted and saturated as other benchmarks,” he said, pointing to the challenges of designing structures and managing resources within the game’s dynamic world.

A New Frontier for LLMs

Unlike past AI projects that focused on board games and classic video games, today’s large language models (LLMs) can analyze text, images, and other data types, allowing for more advanced interactions. “Games are just other ways you can do decision-making with AI,” said Matthew Guzdial, an AI researcher at the University of Alberta. Games can reveal subtle differences between LLMs like GPT-4 and Claude, which can “feel” different based on how they process and respond to prompts.

Insights and Limitations

Pictionary’s structure, Calcraft suggests, mimics the dynamics of generative adversarial networks (GANs), where a generator and discriminator model collaborate and compete. In this case, the “drawer” attempts to convey a concept while the “guesser” interprets it, requiring elements of strategy and collaboration between the models. “The best one to draw is not the most artistic, but the one that can most clearly convey the idea to the audience of other LLMs,” Calcraft noted. However, he acknowledged that Pictionary is still a “toy problem” and doesn’t equate to practical AI applications but could highlight spatial and conceptual understanding in AI.

On the other hand, Singh sees Minecraft as a tool for testing an AI’s reasoning ability, observing how certain models align with his expectations on reasoning tasks. Yet, other experts are skeptical. Mike Cook, a research fellow at Queen Mary University, believes that while Minecraft may resemble real-world activities, it doesn’t necessarily improve problem-solving capabilities. “From a problem-solving perspective, it’s not so dissimilar to a video game like Fortnite, Stardew Valley, or World of Warcraft,” Cook said, adding that AI systems still struggle with adaptability in new environments, and struggle to solve unfamiliar problems.

Looking Ahead: Games as a Lens into AI Development

Testing AI through games like Pictionary and Minecraft underscores a larger trend in AI research, moving beyond static benchmarks to dynamic, interactive testing grounds. While these games may not fully simulate real-world challenges, they offer insights into how AI models manage uncertainty, learn from interactions, and approach creative tasks.

Games might not replace traditional benchmarks, but as developers explore new ways to push AI capabilities, they may reveal where current models fall short—and where the next innovations need to occur.

Editor’s Note: This article was created by Alicia Shapiro, CMO of AiNews.com, with writing, image, and idea-generation support from ChatGPT, an AI assistant. However, the final perspective and editorial choices are solely Alicia Shapiro’s. Special thanks to ChatGPT for assistance with research and editorial support in crafting this article.