Question: In a multiple choice question should my choices be in alphabetical order?

Answer: I only heard this “rule” recently – and I don’t see any reason for it. I suppose it would lessen the possibility that you put the correct answer as “C” all of the time, but we always encourage our clients to use the scramble choices option in our system — which would have the effect of messing up the alphabetical order anyway. The only “order” rule I know of on choices is that numeric choices should be in ascending order.

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Item Grouping

Question: Does it make sense to group items in a test by “type?”

Answer: That’s not a bad idea, particularly if you can group them within sections. It’s a little less confusing to the student. If you intermix multiple choice items with “All That Apply” items there is the possibility that the test taker may not notice that some of the questions may require multiple responses (though it’s always a good idea to state this in the question stem).  If you segregate different question types by section you can have the section introduction provide instructions: “In this section each question may have more than one correct response. Please be sure to select ALL of the correct answers.”

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Question: Is it OK to create questions of the type:

Question Stem
  a. Choice A
  b. Choice B
  c. Choice C
  d. A&B
  e. A&C
  f. B&C
  g. A, B and C

Answer: I really don’t like these questions. First, I think they are confusing to the student, especially if there are a large number of possible choices. Secondly, they are an artifact of paper-based testing where there could be only one single correct choice. Since online testing systems easily accommodate multiple response questions just have the first three choices and ask the student to check all the choices that are correct.

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Question: What difficulty level makes a question “hard?”

Answer: First let’s review: Difficulty levels are a measure of the percentage of students that got a question correct. So a difficulty of 1 means 100% of the students got the question correct and a difficulty of 0 means everyone got the question wrong (the opposite of how you might think the scale should go). In a test, difficulties are both absolute and relative. What do I mean by that? Clearly a question with a difficulty of .95 is an easy question (95% of the students got it correct) and a question with difficulty .20 is a difficult question (only 20% of the students got it right). (A question can be difficult because it is confusing or poorly written, in addition to just being a hard question to answer.)

But what about a question with difficulty .70? Is that a hard or an easy question? Well, if the average test score is 95% then that’s a hard question (relative to the other questions on the test), but if the average test score is 40% then that’s an easy question (relative to the other questions on the test). So, questions at the bottom or top are absolutely easy or difficult, and questions in the middle can best be judged in relative terms.

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Question: Are there simple ways to detect cheating without doing a complex statistical analysis?

Answer: Here are three simple things you can look for:

1. If your test takers are naturally divided into groups – let’s say a sales force that is divided into districts of ten people each – look for districts that consistently outperform other districts. Either you have a district of geniuses or perhaps they are collaborating on the tests.
2. Look for individuals or groups of people who do better on the more difficult questions (look at the item analysis for this) than the easier questions. This might indicate that someone has provided the answers to the difficult questions.
3. If you give a test to different groups of people over time (for example, a new hire test) and you see rising test scores over time then you can be pretty sure the test questions have become public, and you need to create a new test.

None of these simple methods proves cheating but they should alert you to be more vigilant and perhaps to investigate further.

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Question: I have a compliance test with a passing score of 100%. Can I allow the students to try each question as often as necessary until they get it correct?

Answer: Somewhat surprisingly I get this question a lot. All I can say is: Why not just give each student 100% and save all that test taking time? Anyone will eventually get a multiple choice question correct if you allow him to keep selecting choices until he hits the right one.

Assigning an arbitrary passing score of 100% to any test is not valid, but I understand why this is done for compliance tests. So, give as many tries as you need to, if you must, but at least use a standard testing strategy: Each question gets asked once in the test and if the student does not get 100% he/she must take the entire test again. I recommend that you not give the students the correct answers between test tries; make them figure out what the correct answers are.

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Question: My test reliability is low. Is this a problem?

Answer:For mastery-based tests where the test scores tend to be uniformly high test reliability tends to be low. Why is this? Reliability is a form of correlation and when one measure is pretty uniform the correlation of that variable with any other variable tends to be low. For example, let’s say we want to correlate test score with job performance (predictive validity). All else being equal you would expect high test scores to correlate positively with job performance. But what if everyone scored 100% on the test? Then the correlation with job performance will be zero. Both poor performers and high performers got the same test scores.

So, if your average test score is somewhere above 90%, the variance of the scores will be low, and consequently you are probably going to have a low test reliability. It’s an artifact of the kind of test you are giving.

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Question: How do learning objectives relate to test sections?

Answer: You can think of a test as organized like a hierarchy. Starting from the top down:

• Test
• Sections (optional)
• Learning/Assessment Objectives (optional)
• Questions

So a test could just be a collection of questions. (Most of the tests I see are just that.) But if you want to get more information out of your test results you will want to think about adding one or both of the intermediate elements to the hierarchy. Where do your students have weaknesses? Aggregating questions into learning objective scores will tell you that. Learning objectives can be further rolled up into sections, whose scores give you a slightly bigger picture of performance. As an example, let’s say you have a learning system made up of four modules and each module contains a number of learning objectives. So when you create your test, associate the individual questions with their respective learning objectives and divide the test into four sections, each section corresponding to a module. Then in your test results, in addition to having an overall test score, you will have module scores and learning objective scores. You can use this information for remediation, to guarantee mastery and to improve your training programs.

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Question: If a multiple choice question carries one point should a multi-part question with four choices (e.g. a Select All That Apply question with four possible “correct” choices) carry four points?

Answer: Yes, that makes sense, but you also have to remember to give partial credit — otherwise the student could get three out of four parts correct but still be given zero credit for the question.

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.

Question: If I give a test, score it and then realize I made a mistake on one of the questions, should I rescore the test?

Answer: Yes, for reasons of both validity and fairness. But how you rescore depends on what kind of mistake you made… and your tolerance for dealing with some upset students. There are actually three cases:

Have a question only an expert can answer? Let us know! If you want more detail, disagree, or want my rationale for an answer, feel free to comment.