It seems every teacher who has worked in Alaska's rural school system has a story of the cultural ignorance of standardized tests. A question that stumped a student in my wife's 2nd-grade class asked for the best choice on how to get to a hospital: boat, ambulance, or airplane. Since the nearest hospital is 300 mile away, the student circled the logical, yet "incorrect" answer: airplane.
The problem isn't that the question is "culturally biased." The problem is that the person who wrote that question is as lazy as they are incompetent.
I've spent a lot of time writing multiple choice questions for quizzes and exams during my time as a philosophy graduate student. It's something of an art form for me. And the number one principle that I adhere to is that a multiple choice question should have a clearly and unambiguously right answer. There should be no wiggle room: any interpretation that might make one of the other answers even arguably right should be eliminated by the structure of the question. Here's a good example of a bad question:
William Ockham thought that a term in spoken language represents real objects:
(a) Alternatively
(b) Primarily
(c) Secondarily
(d) None of the above
There are numerous problems with this question. First off, it's asking what someone who is dead was thinking, which is a stupid thing to do, and it doesn't even provide a timeframe for that thinking, which might have changed over the course of Ockahm's career. Even if it were the "correct" answer (which it isn't) "None of the above" could arguably be a poor choice here because in selecting it, you're endorsing the idea that terms in spoken language represent real objects, which isn't necessarily the case. (In fact, it is the case here, but only contingently.) There are at least two other things wrong with this question, but they're minor quibbles at best.
Anyway, here's the question, written much better:
In the first book of his Summa Logicae, William Ockham states that when a term in spoken language represents real objects, it does so:
(a) Alternatively
(b) Primarily
(c) Secondarily
(d) The question includes a false premise: Ockham denied that spoken language represented physical objects.
Now some "test bias" is real; the old stand-by of the tale of the "Regatta" question is a good example of that. But a vast amount of it is, I think, just awful test-writing.
This is particularly on my mind because I had cause to run into a really, really awful test question just the other day. I'm reproducing this from memory, so some of the minor details might be wrong (in fact, I'm rounding the numbers to make it easier to solve in your head). But I'm confident I've got the structure right.
Mary is taking a road trip. She will be driving from San Diego to Marin County, a 520 mile trip. She plans to refill the tank when it is only 1/4 full. She will travel at an average speed of 50 mph. Her car gets 25 miles per gallon average gas mileage, and has a 20 gallon tank. Assuming she starts with a full tank, how long can she drive before she has to refill?
a) 5 hours
b) 10 hours
c) 7.5 hours
d) She won't need to refill
This is an terrible question. What does it mean when it asks how long can she drive before she has to refill? Does it mean how long till her tank is empty and she MUST refill? Or does it mean how long until she reaches the point where, according to her plan, she SHOULD refill?
On a purely technical level, it must be the former (and the answer should be (b)), because that's what "how long can you drive before you have to refill" actually means. But in the context of the question, it seems obvious that the question's author wants an additional step in the calculations -- how long till she gets to a quarter-tank, in which case the right answer is (c). In fact, I'm inclined to think (and I marked down) the correct answer is (c). My job on a multiple choice test, after all, is not to get the right answer. My job as a test-taker is to get the answer that the test designer/grader thinks is the right answer.
After all, why would there be useless information like that bit about her plan in the question, right? Well... why would there be useless information like the names of the cities? Questions have useless information in them all the time.
My point is just this: a lot of test questions really are no good. And we shouldn't necessarily scream "bias" when, in fact, the problem is incompetence.
No comments:
Post a Comment