The Reeve Guildmaster's Art: Making Good Tests
[01/24/2006] [Matthias]The burden of learning the new rules is arguably the greatest on Guildmasters of Reeves since they have to give the tests that certify would-be reeves in Amtgard. If GMRs fail to give good tests, we may end up certifying unqualified reeves or hinder players from serving who would otherwise make be good reeves. It therefore falls to the GMRs to give the best tests they can. But what makes a good test? A good test will act as a set of honest scales from which a reeve can be weighed and found to be worthy - or found wanting. This means that a test should be "impartial, fair, have a good eye, and be well-versed in the rules." Each of these points can be applied to the design of a good test. Test Impartiality Impartiality can be a tricky thing. Every GMR belonging to a group with rules clarifications is tempted to include questions on those clarifications in their tests. While this can be quite practical in the sense that you'd want your local reeves to be familiar with the "house rules" of your group, it is not good test-writing. A reeve's test is about one's knowledge of the official rules of play, not any one group's modifications. A reeve candidate should be able to take a test written by a GMR from Dragonspine, Neverwinter, the Burning Lands, or Rising Winds and, if he passes, enjoy the same level of public trust regardless of whose test he took. One of the strengths of playing in a national LARP such as ours is that we all play by the same set of rules and that we can assume everyone else does too. If a test contains questions geared to any one group's playing style, that solidarity is harmed. Therefore, an impartial test should be universally portable to any Amtgard group. Test Fairness The fairness of a test is derived from the fairness of the test questions. To be sure, no test question is perfect and some can exist in a gray area where the test question is fair for some candidates and not for others. Most tests can survive having a few unfair questions, but if those unfair questions are in a large enough number that it could make the difference between passing or failing the test, then the test is an unfair one. There are at least six ways in which a test question can be unfair: 1) The unfair question is out-of-scope for the test. A test question must be specific to the Rules of Play. For example, if a test question asks about the level abilities of a monster that appears only in the Dor Un Avathar and not the rulebook, or about the awards that a regent can bestow, it is an unfair question. The test is supposed to test a candidate's knowledge of the Rules of Play, not the monster handbook or the corpora. 2) The unfair question is too complex. While not specifically mandated, a test is best if it can be taken in a timely manner. No reeve candidate should be asked to take a test outside of normal Amtgard time unless it can't otherwise be helped, nor should a candidate have to spend hours taking it. If the test cannot be completed within 30 minutes to an hour, it is an unfair test. This means that a question requiring the candidate to write a theme paper or a thousand-word essay on some rules-related topic is an unfair question. 3) The unfair question is too obscure. There are an infinite number of obscure questions on the Rules of Player that may be devised. For example: "list the Nth word that appears in the main text on each page of the rulebook, not counting headers and section titles, and put them in reverse alphabetical order." While such questions are technically within the scope of the Rules of Play, if a candidate has to have memorized the rulebook word-for-word in order to answer the question, it is an unfair question. That is an extreme example, but the fairness of such questions can get a little iffy. We are not testing reeves to see if they have photographic memories; we are testing reeves to see if they know the rules well enough to serve as a reeve. It doesn't make much sense to ask about very rare rules that the average reeve will mostly likely never encounter more than once or twice a year (although such questions are quite useful as bonus questions). 4) The unfair question does not contain enough information. A test question needs a certain amount of context which will let the test-taker determine what information is being asked for. For example: "What happens if one player does something to another player? Is that legal?" While a bit silly, that example demonstrates the fact that not enough contextual information can make a question impossible to answer. 5) The unfair question deals with a rule having multiple interpretations. It a hobby of some to hunt for loopholes or gray areas or unclear wording for purposes of exploitation and gaining an unfair advantage on the field. Therefore, it is important for test-writers to make sure their test questions are not susceptible to creative interpretation. No test can be made foolproof, but one has to do one's best. For example, under 6.0 Rules of Play, the question "Does a lightning bolt striking a player's weapon kill the player as well?" was an unfair question since the ambiguous wording of the effect of the Lightning Bolt spell in 6.0 led to some individuals believing the answer to be a "yes" and some believing it to be "no", all with complete intellectual honesty. 6) The unfair question is misleading. It is the habit of sadistic or playful people to incorporate so-called trick questions into their tests. Such practices are useful for testing to make sure test-takers are paying attention, but their overuse can lead to a candidate becoming frustrated or confused. It may also compel them to take a meticulous approach to completing the test as they scan for the next trick question. Such trick questions can be fair or unfair; it mainly depends on the nature of the subtle error that makes the item a trick question. A fair trick question might contain a subtle error pertaining to a rule common enough that anyone who overlooks it really should have known better. An unfair trick question is one which would require the reeve candidate to read the test writer's mind or to otherwise be able to predict what he was really thinking in order to sidestep the land mine. The practice of incorporating trick questions into one's test is not a good thing to do unless done in moderation. A reeve test is not, after all, meant to test whether a candidate can find and avoid falling into all the traps laid by the trick questions; the test is intended to determine whether a candidate knows the rules. Therefore a question such as, "If a Druid charges a flesh-to-stone magical ball and tosses it at a 3rd level Monk, is the Monk turned to stone?" is an unfair question if a "yes" answer is identified as wrong simply because Druids now call that particular magical ball Petrify and not Flesh to Stone. "Having a Good Eye": Test Elegance While it may not seem that important at face value, a good test should be asthetically pleasing and easy to browse. Any experienced publisher will tell you that using very small type or using an odd font or a font that is sans serif can interfere with the comprehension of the text. Most people don't notice such details consciously, but having to fight the text can make reading and comprehension more difficult or more tedious. Asthetically-pleasing tests don't have to have illuminated characters or incorporate pretty pictures unrelated to any of the questions, but a pleasant and legible font more than does the job. Additionally, the test should be written using decent grammar and in a style that would not confuse or mislead the average literate test-taker. If the test writer has very poor writing composition skills, it's his responsibility to find someone who can edit his test and proofread it for legibility. The test should also have a structure which changes pace occasionally. Everyone can attest to having to trudge through an endless series of multiple-choice or true-false questions, losing their place, or succumbing to periods of zoning out. A good test should change its tempo once in a while so that the test-taker can properly maintain their concentration. There are several common types of questions which most individuals will recognize immediately. There are advantages and disadvantages to each: 1) True/False. This is the most simple type of question, and sometimes the most insidious because they are easiest to use as trick questions. However, their chief disadvantage is that a test-taker guessing randomly will guess the right answer 50% of the time. A test that is 50% True/False questions and 50% everything else can open the door to candidates passing the test by randomly guessing on the True/False section and getting a perfect score on everything else. 2) Multiple-Choice. This is probably the second-simplest question type since there are a finite number of answers to choose from. Multiple-choice questions make it more difficult to improve one's score by random guessing, but an intelligent candidate can often use logic to out-think the question and guess the right answer if they weren't otherwise certain of it to begin with. Because of this, it is a good idea to include a "none of the above" answer for each multiple-choice question. The main advantage of this type of question is how quickly the test-giver can grade the test, with a secondary advantage of not having to decipher much bad penmanship. 3) Matching: This is a variation on the multiple-choice question type, the difference being that each group of questions uses the same set of possible answers. It is by no means required that every choice per group be used, nor should a given choice be necessarily used only once. A group of matching-type questions with a one-to-one ratio of possible answers is fairly easy to break, where each question has a single corresponding answer and there are no extra answers. In such cases, a test-taker can skip a question and return to it, and be able to give the right answer simply because it was among the ones left over. The key to making it hard for the test-taker to out-think the test in this fashion is to include several unused answers similar to some of the right ones, and to include some questions that will reuse one or more of the possible answers. That way, no one-to-one ratio exists. 4) Short Answer: This type of question is good because it gives nothing away to the reeve candidate about the answer to the question other than what is written in the question itself. There is no list of potential answers that can be used to out-think the test and discover the right answer. Probably the biggest disadvantage of this question type is that the test-giver may have to deal with illegible writing. Otherwise, this is usually the best type of question to use. 5) Fill-in-the-blank: This is basically a variation of the short-answer type. As a side note, it isn't a good idea to use fill-in-the-blank questions to quote the rulebook directly since there are some passages which Amtgarders can recite by heart because they've read them often enough to have memorized them. A test is not meant to test a candidate's ability to recall information by rote, but rather his comprehension of the rules. 6) Listing: Listing questions are quite useful since no clues or hints need be given away for free. Listing questions usually require the candidate to be familiar with several different parts of the rulebook in order to give all the answers required. For example, answering the question "Which classes can use javelins?" requires that the test-taker be able to answer 'yes' or 'no' when he asks himself about each class in turn. Extrapolating from what he remembers about the allowed weapons of each class, he can then draw up a list of which ones are allowed to use them. 7) Essay: This question type, while popular with college professors, does not really belong on a reeve test. On a test written to certify players as referees for a game, essay questions go a little overboard, except perhaps in extraordinary circumstances such as a test given to a person who needs to qualify for a kingdom-level office. Even on kingdom-level reeve tests, they should be used conservatively and possibly only for extra credit. A good reeve test that varies its pace might have ten true/false questions, five multiple-choice questions, a number of listing questions totalling fifteen answers, ten short-answer questions, a group of ten matching questions, and ten more true/false questions. Sixty answers in all, a nice and round number making a passing grade (75%) exactly forty-five items to be answered correctly. "Well Versed in the Rules": Test Comprehensiveness Good tests cover what need to be covered. One that only tests for knowledge of the basic combat rules, weapon construction, and the rules for reeves will certify reeves who can be trusted to oversee tournaments and ditch battles, but there will be no guarantee that they can reeve class battles or quests. A reeve test should therefore cover every aspect of the Rules of Play. There are fourteen broad categories which should all be addressed to some extent in any good reeve test: 1) The rules for reeves. Every reeve test should address the duties and responsibilities of being a reeve. This category must never be neglected. 2) The basic combat rules: what constitutes a hit from a melee weapon, what constitutes a hit from a ranged weapon, how hits on garb are adjudicated, the legal strike areas, deaths and lives, subdual blows, etc. 3) Weapon construction: safety standards for weapons and shields, the various lengths and sizes of weapons and shields, the construction requirements for great weapons, etc. 4) Projectile weapons: maximum poundage and draw length for bows, the standard for longbows, the bow-to-player ratio, the construction requirements of arrows, etc. 5) Armor ratings: which armor types are worth what values, armor guages, safety requirements for armor construction, armor value modifiers, etc. 6) The rules for earning credits and gaining class levels. 7) Basic class information: garb requirements, allowed equipment, rules for playing the peerage classes, the rules for playing Raider and Peasant, the spellcaster-to-player ratio, etc. 8) Class abilities: the functions of the more common and problematic class abilities such as Berserk, Sanctuary, Steal Life, Fight After Death, the specialty arrows, etc. 9) The magic-points system and buying weapons: how points are gained by levels, how weapons are bought by spellcasters and at what costs, how magic points are redistributed after buying weapons, etc. 10) The basic rules of spellcasting: how spells must be cast, what constitutes an interrupted spell, the physical limitations on being able to start casting magic, the rules for counts and spell ranges, etc. 11) The rules for enchantments: casting them, dispelling them, carrying them, how the immunity enchantments work, how the armor enchantments work, etc. 12) The specific magics: The incants, effects, and other information concerning the most common spells, magical balls, enchantments, and neutrals. A few fixed enchantments should also be randomly addressed. 13) The monster rules: who can play a monster and how often and at what level, how to become a monster via the Reincarnate and Transform enchantments, the statistics of those monsters which appear in the rulebook for that very purpose, and the monster-to-player ratio. 14) Miscellaneous stuff and anything else that does not fit into one of the above categories such as terrain rules, the different battlegame types, the rules for personas, the garb privileges of knights, etc. Any reeve test intended to test a player's knowledge of rules should have at least a handful of questions representative of each of these fields. None should be left unaddressed. Conclusion These ingredients, put together, are what distinguishes a good reeve test from a bad one. It is incumbent upon every GMR to make sure his tests are impartial, fair, elegant, and comprehensive. Extra Credit! Finally, a few good bits of advice related to test composition: * Make sure that no one question could possibly give the answer to another question elsewhere in the test. A well-designed test will minimize the ability of a test-taker to use guesswork to discover the answer to a question he could not have given without taking the test in the first place. If a test-taker knows that the scout sash color is either green or brown but isn't sure which, and there is another question that hints or states that the druid's sash is brown, he will be able to give the right answer for the color of a scout's sash if it should appear elsewhere in the test. * If you are including bonus questions, make sure that extra credit is also rulebook-related. Don't include questions on the corpora or the Dor Un Avathar. If you include a set of bonus questions, apply the same standards as you would any other question, with the exception that bonus questions are meant to be harder or more obscure than normal questions. * Don't allow test-takers to write their answers on the test sheet itself, but require them to fill out their answers on a separate sheet of notebook paper. This will let you reuse copies of your test and save money on photocopying or printing. Just be sure you include the admonition of not writing your answers on the test paper so the test-takers will know not to (or at least have no excuse if they do it anyway).
[ discuss on forums ] | 
