Players use books differently from authors / designers, a point which is
obvious, but I’m trying to figure out how to phrase the question that
suggests. Maybe something along the lines of:

How easy was it to find information? When you had a question about how something worked, could you find an answer in the text? Easily? A clear answer? An answer that made sense and was satisfying? (I’m thinking of Polaris here. Whenever we weren’t sure how something worked, the answer was right there, in the text. Granted, it was at the published stage, not playtest stage, but that’s still something we find far too rare. I’m also thinking of Kerberos Club Fate Edition, which I love to pieces and am running, but I have had to create a couple of spreadsheets because finding certain pieces of character generation information, whether for PCs or NPCs, involves way too much flipping pages.)

Is it done yet? If not, what do you think needs to be done? (That may be asking for trouble — I’m not sure. The only games I’ve written are larps which must be done by runtime. And, the answer to what I think needs to be done generally involves “Please let me give a full once over to what you think is the final, ready to go text, preferably in the format buyers will be seeing.” This may not always be practical. A playtester isn’t the same as an editor, but there is some overlap. Also, in a playtest, I may not know the finished game won’t have an index — or that it will. You might not send your index out with an early playtest version, and I may simply assume that, of course, there will be an index in the finished game.)

I’m not sure how to phrase this, but it’s some variant of “Are you clear on what the rules are, given I’ve just made a major revision to the game / made multiple revisions, and I may have caused whiplash?” But, I suspect the general “Are the rules clear / What isn’t clear” variants still work.

My biggest question of a playtest GM: what did your players disagree over the meaning of? (IE, what were the sections the rules lawyers were able to twist?) 

One thing that I’ve found to be very important is to be able to stop play and change a rule.  I think it’s best to wait until a full round has completed, i.e. back to the starting player’s turn, then explain the new mechanic.  If something isn’t working quite right, this is a great way to get more value out of a test session.

I also find it helpful to play two rounds face-up to familiarize everyone with possible card play combinations, then reshuffle and play normally.

And finally, I frequently have a pre-printed survey.  I’ll have things like a sliding scale between random and skill based so that I can see if I’m hitting the sweet spot that I’m looking for, it also lets me collect email addresses for potential mailing lists and names to include in tester credits.

Another important one is, when the players engage with the mechanics, are the results appropriate to the setting? Does your Conan homage reliably produce mighty-thewed barbarians? That’s good. Does your Sherlock Holmes homage reliably produce mighty-thewed barbarians? That’s bad.

Did any rules not make sense?
Was the significance or purpose of any rules unclear?

Did you ever feel like a rule prevented you from doing something exciting in the game?

What was your favourite part about the game?

(Aside from those, I mostly ask very specific, contextual questions about a particular design.)

http://dl.dropbox.com/u/692162/FateTacticalCombat.pdf