Home


Search

Back issues

Guestbook

Community

Related links

Subscribe

FAQ

XYZZYnews

Oh No! Beta!
Tips and Techniques for
Beta-Testing Games-in-Progress

by C.E. Forman

Any author who has written a work of IF of medium size or larger should be wholly familiar with the process of playtesting the game, discovering bugs and parser problems, making certain that the game is as clean as is reasonably possible before its official release to the IF archives. The key issue, though, is whether the game's testers themselves possess the same degree of familiarity.

The beta-tester's goal is not to play a game solely for leisure and escapism, but to willfully risk his or her own spoilage of the fantasy, with the intent of using it to prevent such disastrous results from reaching a broader audience. This is the price to be had in exchange for becoming a part of another author's world and exerting a certain degree of one's own influence upon the final outcome.

Testing, in its own way, is as much an art as the creating of the game world itself. To playtest effectively, one must come to understand the link between tester and game, and further, between game and author, in order to draw across the third and final gap between author and tester. Bridging the gap is the most mutually constructive manner is the goal of the IF beta-tester.

Offering Your Service

Most game authors recruit the assistance of playtesters via a Usenet post, or by personal e-mail inquiries to testers whose names they are familiar with (possibly through recommendations from other authors). It is important at the outset to establish the amount of time you, as a tester, will be able to spend on the project. If you don't think you'll have much time (or any at all), either warn the author ahead of time so s/he can make arrangements with other testers, or don't sign up at all. If, once you've begun, it turns out that you have far less time than you'd anticipated, let the author know as soon as possible so that replacement testers can be found. Under no circumstances should you leave the author hanging as to whether or not you're still with him/her.

Structure of the Playtester's Report

My typical beta-testing report is a standard plain-text file, divided into four sections, with a header at the very top identifying the game, the report number (numbering reports makes it easier for both authors and testers to identify and organize later), and the beta version the bugs came from. You may want to include your name here as well.

These four sections serve as a good method for breakdown and classification of bugs:

  1. Programming bugs. These deal with significant errors found in a game, be they logical errors or outright crashes.
  2. Parser problems. These consist of minor annoyances you've had when trying to select verbs, nouns, phrasing, et cetera.
  3. Cosmetic bugs. Typographical errors, extra line-feeds and spaces, and the like. These are not true errors, but are important to the overall appearance of the game.
  4. General feedback.
The first three sections should consist of appropriate segments of a game transcript (beta-testers should _always_ keep transcripts) cut-and-pasted in, with the unnecessary steps edited out to reduce clutter. If the bug in question can be explained in only a few words, a transcript segment is not necessary. When using my own words rather than the game's, I prefer to surround comments with square [] brackets to make it easier for the author to identify where the text is coming from. In any case, include a line of dashes or asterisks to separate one bug from the next, to keep the report easy to read. Also, when dealing with typos, it's helpful to use a carat (^) symbol below the offending word to point out just where the error is.

Structuring reports in this format makes the author's job far easier, as it enables him/her to select the type of bugs to correct, based on the author's current mood. This stage of game development can be frustrating at times, and on some days an author simply may not feel up to tackling more complex bugs. It's more reassuring for the author to have an organized group of typos and parser quirks to work at during these times, rather than having to read through a jumbled report and pick out appropriate bugs on his/her own.

The fourth section, general feedback, should serve as a means of input from tester to author, and is used to cover anything that doesn't fit into the first three categories. This is the place for a tester's own personal evaluation, covering any plot holes, comments about the puzzles and overall layout, and general praise and (constructive) criticism. It is of the utmost importance that a tester provide a good deal of positive feedback here as well. Playtesting is the absolute last thing an author wants to do after spending months on writing and coding. By the time a game reaches the testing stage, the author's morale can be at its all-time low, and it's likely to stay there if the only feedback received about the game is a long list of everything that's wrong with it.

Following these guidelines, here's a fictional sample report using some early bugs found in my own game, "The Path to Fortune" (but now fixed):

    "The Path to Fortune" Playtesting Report #1
     -------------------------------------------  
     =====================================================
     1 - Programming Bugs ================================
     =====================================================

     [I can unlock the abandoned shed with any object in the game.]

     ----------------------------------------------------

     >cash
     You're broke.

     >w

     Borthur's Smithy

     >cash
     You have 12 gold Renkin, the standard unit of currency in the

     [I haven't asked for the gold yet.]

     =====================================================
     2 - Parser Problems==================================
     =====================================================

     >make a roll
     You'll have to specify what you want to make.

     [I've tried every verb I can think of, and can't get this to
     work.  Are you sure the code is reachable?]

     =====================================================
     3 - Cosmetic Bugs ===================================
     =====================================================

     >x gold
     Something about the unshaped nugget of gold unnerves you. When
     you hold it up to the light, the nugget sines very brightly,
                                             ^^^^^
     [Should be "shines."]

     =====================================================
     4 - General Feedback ================================
     =====================================================
[...and so forth.]

Make certain that your messages are clear enough for the author to determine the problem, and don't send blatantly derogatory messages of any sort.

Followups to Reports

It's quite likely that an author will elect to give feedback on your feedback, whether to explain his/her reasoning behind a choice of implementation for an aspect of the game, to request further details regarding a bug, or to assure you that a particularly nasty problem has indeed been conquered. In instances where an author needs to clarify a problem, it's important to leave enough of the previous message to allow the author to see what the original issue was. Don't edit the old parts out just to save space! (This is one of the few times on the net where you want repetition for reinforcement.)

Also on the subject of feedback, opinions differ, and authors may not agree with everything you say. They may have very good reasons for not wanting to implement some of your suggestions -- some parser quirks may be more trouble to deal with than they're worth (particularly in the case of first-time authors); some solutions may contradict other parts of the game (which you may not have seen yet). Ask if you're not sure, but don't pressure authors too much -- if they refuse, and offer at least a fairly reasonable explanation, let it go at that.

Discovering Bugs

This is the heart of the playtester's job, and the most difficult aspect to learn to do effectively. It's also the most difficult to put into words, so here is a set of common methods I use to break a game:
  • Scanning text. Personally, I can't stand reading my own text. As a result I don't look very closely for typos and extra spaces, instead relying on playtesters to hunt them down. Read all game text carefully.
  • Scenery. A player should be able to examine almost everything the game mentions in the text.
  • "Reversing the solution." It's quite common for an author to unintentionally implement a one-way puzzle. That is, one in which the solution could logically be reversed or recreated, but no code for it exists. An example of this would be breaking one of the mirrors in "Enchanter," then fixing it with the krebf spell (which works, by the way).
  • "Do It twice." AGT games in particular award points for players repeating the same action more than once. Try solving puzzles twice to be sure.
  • "Parser probing." IF language tools sometimes use two verbs that mean the same thing, yet behave differently ("move" vs. "push" is a good example). Try performing actions using every possible method of phrasing you can think of.
  • Containers and surfaces. Objects that can hold other objects can cause a lot of trouble (as evidenced by the famous (infamous?) axe-in-the-bucket bug in Unnkulian Underworld). Make sure that players can't sneak objects into places they shouldn't be able to through the use of a container or surface, and make sure that all such objects only hold the size and number of items they would in a "real world" setting. (Large containers used for inventory management, such as the rucksack in "Curses," are exempt from this last rule.)
  • Animate characters. Even simple NPCs tend to have potential for bugs, so try out all the major NPC interactions ("kiss," "kill," "show," "give," "tell," "throw," "ask about" and "ask for"). Make certain that you give each character an order or two as well.
  • Flammables. Adding a torch or matchbook to an in-depth game can geometrically increase complexity. A default message is commonly put to use for most items, but many times special responses will be needed. Pay particular attention to effects such as lighting a match with a torch, vs. a torch with a match, and lighting one torch with a second.
  • Liquids. IF liquids should be drinkable (or have a special message if you try), and able to be spilled out onto the ground, poured into other containers (assuming they'll hold liquids), and so forth. Check to confirm that a liquid can not be carried in a player's hands.
  • Multiple-step puzzles. Recent games no longer resort to the "one object = one puzzle" format of early IF. It is often necessary to solve problems via a step-by-step process. As a tester, make certain that the objects' states change when one part of the puzzle is solved, to avoid nonsensical repetition. An example of this type of bug might be a vending machine that drops a bag of chips into a bin after a player inserts a coin and pushes a button. Check to make sure that, if the player pushes the button again, the text for the chips does not appear a second time.
  • Daemons. Puzzles involving timing are the most complex to implement. Test for redundant actions -- doing the same thing twice to start the daemon twice. Also make sure that the daemon in question only prints text when its object is in scope to the player (i.e., a player shouldn't be able to see or hear a car running after s/he's left the room it is in).

As a final note, keep your old reports and re-check all the bugs later, after the author has sent an update. Make sure the bug fixes work without causing further bugs, and that they don't disrupt the overall logic of the game in any serious way.

Playtester Ethics

Finding bugs is not the only concern of a beta-tester. Believe it or not, the job can also raise concerns about proper ethical behavior. Game authors put a lot of trust in their testers, and it's important that the testers not betray that trust. Keeping that in mind, here are some simple (if you think about them) guidelines that I adhere to, and that I advise other testers to follow as well:
  • First, and most obviously, never distribute a game you're playtesting. If the author wanted this, s/he'd have released it already, without your help.
  • Keep your playtesting experiences confidential. It's okay to confer with other people beta-testing the same game, but once the game has been released, don't tell players about bugs you found in the beta versions. Authors don't want to appear sloppy or stupid to the gaming public, and indiscriminate discussion of bugs that have already been fixed can make them feel that way. (Again, let the author initiate this sort of talk if s/he wants to.)
  • Don't write zine reviews of games you've beta-tested. Like the authors themselves, you've probably been working too close to the project to give an effective, unbiased evaluation.
  • Don't give out hints over Usenet to games you've tested...at least, not for awhile. Out of necessity, you've already seen the solutions, but other players have not. Give them time to figure things out on their own and to help each other out first, rather than spoiling the game and the author's hard work right at the outset. (By the same token, wait a good time before writing a walkthrough or hints for a game. And even then, make sure you have the author's consent.)

Conclusion

Playtesting can be a very painful process for an author...but it doesn't have to be. With a little extra effort on the part of the playtester, it can be an enjoyable and rewarding experience for both parties.


[right arrow] Go to the next page in this issue
[left arrow] Flip back to the previous page
[top arrow] Go to the XYZZYnews home page
This site is maintained by Eileen Mullin
Legal information