Ghosts in the Procedural Machine

Given the choice, most people would rather have their own unique adventure than play through the same story with the same twists as everyone else’s tale. Games like Spelunky and Minecraft push the boundaries of what is possible within the emergent systems of a video game. The power of procedurally generated content is being harnessed to make everything from realistic texture maps to entire universes. However, procedural generation is, by no means, a magic bullet. There are extreme challenges in developing and testing systems with nearly endless possible permutations.

The perks of a game with procedural generation are obvious to players, but designers would also prefer to write an algorithm that generates endless player possibility than they would create hundreds of levels by hand. This is often the case for small, indie developers with limited time and resources. Brad Johnson is the developer of tile based dungeon crawler, A Spire to the Gods. He says, “As a solo developer, choosing to make levels procedurally has been a life saver, a mood lifter, and a game enhancer. I don’t have to worry about how to come up with interesting level layouts. If I had a team of developers working on levels, I’d probably flesh out the level editor and make custom levels, but that’s only in an ideal situation with unlimited time and budget.”

Whatever the developer’s motives for using procedural generation, it can be a nightmare for quality assurance testers. Uncommon cases arise when you introduce the requisite elements of randomness into the code. Kenny Lee, worked on the hit rogue-like, Rogue Legacy. “Debugging Rogue Legacy was the worst experience I’ve ever had in anything I’ve ever coded. The nature of random generation means some things only happen a certain percentage of the time. If a bug happens to crop up only when say, a particular Room X is placed next to Room Y, you’ve got like a 0.01% chance of it happening. That makes it virtually impossible to track down and fix, but it’s not low enough for players not to encounter.” These challenges explain why many of these titles begin life on early access platforms. A public beta test can be instrumental in discovering rarely occurring bugs.

Of course, the arduous debugging process of a procedurally generated game at least offers variety. Redshirt developer Mitu Khandaker says, “Part of the challenge of developing a game like this is tracking down when there’s a bug because you don’t necessarily know all of the minute possible quirks. On the flipside, it makes it sort of fun to stare at it for two years like I did. Scenarios will come up which I wasn’t anticipating. It’s like, ‘Oh hey, that’s kind of amusing.’ Instead of refining the same content over and over again.” Many developers talk about similar amusing scenarios.

Unexpected combinations often arise in these early testing stages of a procedurally generated game. Sometimes this leads to absurd or interesting outcomes. For instance, Spry Fox’s Daniel Cook describes a quirk from the excellent procedural puzzler, Road Not Taken: “There’s a bug where your spouse will start duplicating. It is like some strange Stepford household.” At a talk given at Rezzed in 2013, the creators of Sir, You Are Being Hunted mentioned a problem in the early implementation of ravens in the game. The trees were generated too close and the ravens were too sensitive to sound. So, if a player startled the birds in one tree a whole chain of birds would be triggered by the wing flaps of one tree after another; the resulting waves of birds were huge, island-wide, upheavals.” Seemingly countless developers have stories of rooms generated with no exits or no way for the player character to move forward. The developers of Xbox One exclusive, Below, encountered a random level with so much fog that the player could hear the sounds of traps and enemies, but they couldn’t see the level.

Dwarf Fortress is a procedurally generated game so ambitious that it seems to constantly create quirky interactions and strange bugs. It is a partially rogue-like game with a sprawling list of intricate mechanics and systems. The game’s development log has become somewhat infamous. Rock, Paper, Shotgun says that it’s “like reading updates from a deity as it puts together a new planet.” As of the date of this story, the Dwarf Fortress development blog lists upcoming bug fixes and tweaks like, “Various evil plots realized in play through regular AI actions” and “Musical instrument use, dancing, storytelling, etc.” The game’s rolling updates create new weird scenarios like these all the time.

Some of the most unique kinks in procedural systems come when the code lacks the context that a human programmer brings to the table. Kate Compton, at UCSC, has encountered many of these issues. She is developing a story generation language called Tracery:

Tracery knows “nothing” except what you’ve programmed in. So it doesn’t have our assumptions, especially the ones that we don’t notice, because they’re so obvious. One time I was trying to write a 1001 nights generator, and wrote “The grand vizier espied the #relative# of the court #occupation# and fell madly in love,” with the goal of having a “kidnapping the gardener’s daughter” narrative. But it came up with “the court architect’s grandfather,” because it doesn’t know that the grand vizier *ought* to fall in love with a young woman, it’s just as happy picking out the elderly grandfather love interest.

A version of Tracery was utilized in her ProcJam 2014 entry Eternal Night Vale.

A similar situation happened during the development of the cyberpunk shooter, Black Ice. The random name generator “didn’t know” that the combinations it was generating created hilarious titles for corporations. Garrett Cooper says, “I think some of the more ridiculous names were ‘Baconfist Hardware Incorporated’ and ‘Moonpeople Orbital Holdings’. Silly things like that pop up. ‘Bladefist’ was another cool one.” Cooper explained that fans of the game created a Steam community page with many absurd examples.

All of these funny stories emphasize the design and testing hurdles to implementing procedurally generated content. It can almost sound too vast to reign in and debug. That is why some companies have created elaborate systems specifically to bolster the testing process. Hazel McKendrick has spoken at length about the systems that Hello Game has created to test the distributions of procedural content across the vast universe of No Man’s Sky. In her ProcJam 2014 keynote, she said, “We have bots that go out and take pictures of the planets.” These bots return convenient images of the flora and fauna generated by the latest code implemented in the game. All of these previewing and testing systems allow the team to generate and iterate on such an epic scale.

Similarly, the team at Amplitude Studios created special debugging tools for testers to tackle the possibilities of Dungeon of the Endless. Lead programmer, Sébastien Dubois explains:

…pretty early on we developed debugging tools for the dev team and the testers. It’s basically a set of ugly texts and buttons overlaying the game interface. It allows us to use “cheats” like forcing the next event type, jumping to another level, playing in god mode (unlimited resources, invincibility) or other crazy things like changing the speed of the game. This is very useful to reproduce specific bugs or test a precise situation without wasting too much time getting there.

It sounds like the tool made it considerably easier to tackle the QA monster.

There is little doubt that rogue-likes and procedurally generated worlds will continue to take players to almost limitless worlds of possibility. Smaller studios will likely continue to benefit from the replay value added by this style of content creation. However, they will be taking on a challenging task in testing the many resulting combinations. Data driven programming and innovative tools that easily preview and test seeds, like those being created for No Man’s Sky, will make the testing easier for developers. Ideally, the occasional anecdote of interesting ghosts in the machine or funny bugs will still sneak in from time to time.