Horrible Multiple Choice Questions

People often complain that standardized tests are "culturally biased." In a textbook I've read recently, there's the following anecdote:

It seems every teacher who has worked in Alaska's rural school system has a story of the cultural ignorance of standardized tests. A question that stumped a student in my wife's 2nd-grade class asked for the best choice on how to get to a hospital: boat, ambulance, or airplane. Since the nearest hospital is 300 mile away, the student circled the logical, yet "incorrect" answer: airplane.

The problem isn't that the question is "culturally biased." The problem is that the person who wrote that question is as lazy as they are incompetent.

I've spent a lot of time writing multiple choice questions for quizzes and exams during my time as a philosophy graduate student. It's something of an art form for me. And the number one principle that I adhere to is that a multiple choice question should have a clearly and unambiguously right answer. There should be no wiggle room: any interpretation that might make one of the other answers even arguably right should be eliminated by the structure of the question. Here's a good example of a bad question:

William Ockham thought that a term in spoken language represents real objects:
(a) Alternatively
(b) Primarily
(c) Secondarily
(d) None of the above

There are numerous problems with this question. First off, it's asking what someone who is dead was thinking, which is a stupid thing to do, and it doesn't even provide a timeframe for that thinking, which might have changed over the course of Ockahm's career. Even if it were the "correct" answer (which it isn't) "None of the above" could arguably be a poor choice here because in selecting it, you're endorsing the idea that terms in spoken language represent real objects, which isn't necessarily the case. (In fact, it is the case here, but only contingently.) There are at least two other things wrong with this question, but they're minor quibbles at best.

Anyway, here's the question, written much better:

In the first book of his Summa Logicae, William Ockham states that when a term in spoken language represents real objects, it does so:
(a) Alternatively
(b) Primarily
(c) Secondarily
(d) The question includes a false premise: Ockham denied that spoken language represented physical objects.

Now some "test bias" is real; the old stand-by of the tale of the "Regatta" question is a good example of that. But a vast amount of it is, I think, just awful test-writing.

This is particularly on my mind because I had cause to run into a really, really awful test question just the other day. I'm reproducing this from memory, so some of the minor details might be wrong (in fact, I'm rounding the numbers to make it easier to solve in your head). But I'm confident I've got the structure right.

Mary is taking a road trip. She will be driving from San Diego to Marin County, a 520 mile trip. She plans to refill the tank when it is only 1/4 full. She will travel at an average speed of 50 mph. Her car gets 25 miles per gallon average gas mileage, and has a 20 gallon tank. Assuming she starts with a full tank, how long can she drive before she has to refill?
a) 5 hours
b) 10 hours
c) 7.5 hours
d) She won't need to refill

This is an terrible question. What does it mean when it asks how long can she drive before she has to refill? Does it mean how long till her tank is empty and she MUST refill? Or does it mean how long until she reaches the point where, according to her plan, she SHOULD refill?

On a purely technical level, it must be the former (and the answer should be (b)), because that's what "how long can you drive before you have to refill" actually means. But in the context of the question, it seems obvious that the question's author wants an additional step in the calculations -- how long till she gets to a quarter-tank, in which case the right answer is (c). In fact, I'm inclined to think (and I marked down) the correct answer is (c). My job on a multiple choice test, after all, is not to get the right answer.  My job as a test-taker is to get the answer that the test designer/grader thinks is the right answer.

After all, why would there be useless information like that bit about her plan in the question, right? Well... why would there be useless information like the names of the cities? Questions have useless information in them all the time.

My point is just this: a lot of test questions really are no good. And we shouldn't necessarily scream "bias" when, in fact, the problem is incompetence.

Threats and The School-as-Family

Last week I wrote a little bit about a proposed law governing online threats made off-campus, pointing out what I thought were some ambiguities and incoherencies (is that a word?) in the statute's language. Today I just wanted to rattle off a few brief thoughts about the notion of schools' exercising control of their students' lives while off-campus in general. My apologies... this is sort of jumbled and confused, and I'm too lazy to tighten it up.

What I find really interesting about the thinking behind the move to regulate students' off-campus behavior is that it often coupled with the mindset that schools should be tolerant and welcoming of all cultures, and shouldn't be in the cultural missionary/assimilation business. Yet it seems to me that if you are going to make a moral (as opposed to a legal) argument for being able to punish students for off-campus behavior, you have to accept that one of the missions of a school is to shape its students into a certain type of person, with a certain type of values.

Some of you reading this are probably thinking to yourself, "Of course school is about shaping values. Duh!" But not everyone agrees with this; some reject it vehemently. Still others think they disagree, but what they really think is that schools shouldn't attempt to instill values that they don't believe in themselves.

A private school can kick a kid out for off-campus behavior, because they can have a code of conduct that says, "If you want to be at this school, there's to be none of behavior X, period." (There may be difficulties in enforcement, but that's another matter.) The key difference between a private school and a public school, of course, is that the family is (usually) paying for the student to attend the private school. Attending is an option.

Attending public school is also an option, in the strict sense. But it's also often the only "costless" alternative to meeting a state mandate that your child be formally, officially observed during business hours. In other words, for many families the choice is send the kid to public school or go to jail/pay fines under truancy laws. And that's the reason, I think, that some people -- especially people on the left -- get intellectually or emotionally bothered when public schools get into the character-moulding business: there's no easy way to escape it. The normative commands of the public school are, ultimately, the normative commands of the state, which has a monopoly on force.

In other words, when the public schools can tell you that there's no making online threats, it's really the government telling you that there's no making online threats. And that's probably not a big deal, as long as we're talking about threats, right? I mean, threats are bad, right?

Yet people could reasonably be worried that if a school can ban online threats, then it can expel a student for not going to Mass, or not saying his five daily prayers. Now, you might say, "Wait a second... that's a first amendment issue!" But threats are a first amendment issue, too, and many of the sorts of "threats" that get kids suspended and expelled these days in no way pass the true threats doctrine, which is the test for determining when speech can be regulated by the government because it's a threat. (Typically, a threat is only a true threat if the “speaker means to communicate a serious expression of an intent to commit an act of unlawful violence to a particular individual or group of individuals" and "objective observers would reasonably perceive such speech” to be a threat.) Most incidences of fiction-gone-wrong and online venting about teachers and principals aren't really intended as a "serious expression of an intent" to work violence.

"But we can't be too careful in this post-Columbine world," some people say. Well, yes, actually -- for the purposes of the First Amendment you can be too careful. And it's especially egregious when the speech in question isn't even occurring on campus. The student didn't make a statement on campus; the fact that other students are disrupting the school environment by using their phones and computers to access the students' writing isn't grounds for punishing the student.

My point is just this: either you think that schools are in the character business, or not. And if you do, either you think that the First Amendment actually matters, or you don't. Unfortunately, I think that there are a lot of people out there on the left who claim that schools shouldn't be in the character business, but really think they should, and a lot of people on the right who think that the First Amendment matters, but are willing to toss it under the bus when it doesn't fit their policy agenda.

Some Off-The-Cuff Legal Analysis

So it seems that the Illinois state legislature has gotten it into their head to pass a law that allows schools to expel kids for online behavior. Here's the summary from the legislature:

Provides that a school board (including Chicago) may suspend or by regulation authorize the superintendent of the district or the principal, assistant principal, or dean of students of any school to suspend a student for a certain period of time or may expel a student for a definite period of time if (i) that student has been determined to have made an explicit threat on an Internet website against a school employee, a student, or any school-related personnel, (ii) the Internet website through which the threat was made is a site that was accessible within the school at the time the threat was made or was available to third parties who worked or studied within the school grounds at the time the threat was made, and (iii) the threat could be reasonably interpreted as threatening to the safety and security of the threatened individual because of his or her duties or employment status or status as a student inside the school.

So there are several elements here, and I want to talk briefly about each.

1) There must be an explicit threat.
This is the most interesting, because the incident that is being reported as the catalyst for this law apparently didn't involve an explicit threat. The 2005 Oswego district incident was a student who apparently posted the phrase, "I'm so angry I could kill." Even if what was actually posted was something like, "I'm so angry I could kill Mr. X, that son of a (bleep)", that wouldn't be an actual threat. Maybe if it were said in Mr. X's presence, while you were holding a knife and twirling it menacingly... but then the bulk of the threat would be the knife twirling.

My suspicion is that this law is going to run into problems about what is and is not a threat, and that districts are going to have to resort to a "disrupts the school environment" defense for their expulsions, a la Tinker vs Des Moines and related cases. I predict that these attempts will be successful. In other words, I predict that the "explicit threat" requirement of this law is going to essentially be written out by judicial interpretation and that this will just be a law that extends the school's power over student speech on campus to govern speech on the internet as well -- but I'm a pessimist. Maybe the courts will actually enforce the statute as written and will require an explicit threat.

2) It must be against school-related personnel.
I'm not really sure what that means, other than employees and students -- maybe they're talking about PTA moms or something. Maybe they're talking about guest-speakers who come to campus? No real problems here.

3) It must be made on an internet website
I'm not an Illinois lawyer, but this might be problematic because there's much more to the internets than "websites", at least how that term is used in a technical sense. Presumably this is an easy thing for courts to ignore, though. If it's on the internet, I bet they'll consider it a "website" because it's a site on the web.

4) The site must be school-accessible:
This is an interesting requirement -- that someone at the school (even, presumably, office staff sitting at networked computers) -- must be able to access the website at the time the threat was made. There are three problems that I can see.

First, it's hard to determine when an online threat is "made". Is it when it is typed? When it is posted? When it is read? This isn't like me walking up to you and saying something, after all. What if a threat is posted to a secure website that is not school accessible, and is only accessible to certain non-campus IP addresses, but then those protections are removed? When was the threat made? There's probably case law on this sort of thing, which I'm too lazy to look up. But it seems like a thorny issue.

Second, I wonder if the access from school can count someone violating school policy. Let me explain what I mean. Let's say the threat is posted to a Facebook page that is only viewable by "friends". Let's also say that the school has a strict no-internet policy, but that some students violate that policy by bringing smartphones onto campus and using them. Does that count as a website "accessible within the school"? What if students are not supposed to use school computers for Facebook, but do anyway? Does a student have a right to rely on school policy to protect him from expulsion for making threats?

Third, I'd want to know what "accessible" means. If a page can be reached, but is password protected, is it "accessible"? Does it matter who knows the password? Does it matter that the password can be hacked, or that doing so is a violation of law?

I'm imagining that something like the following rule is going to be followed: if it shows up on someone's computer or wireless device, it must have been accessible. But you might think that a student should be able to tell whether his or her behavior is going to get him or her expelled before some super-hacker at school goes to work on the website in question.

5) The threat could be reasonably interpreted as threatening to the safety and security of the threatened individual because of his or her duties or employment status or status as a student inside the school.
I won't lie... this last bit has me confused.

Presumably, if we're taking the "explicit threat" portion of this proposed law seriously, there's already a requirement that the threat "threaten the safety and security" of the threatened individual. So that can't be what this section is about; the important part really has to be the "because" clause. But I have absolutely no idea what it means for someone's safety or security to be threatened because of their relationship to the school. The most common-sense interpretation is probably a sort of motive-analysis: is this threat being made because the threatened person is a student or employee?

But the answer to that is almost always going to be "no". Threats get made because people do things that make people angry, not because someone happens to be a student or a teacher. So is the real question whether the underlying antagonism had its genesis in the schoolyard, or in the principal's office? So maybe it's not OK to threaten the teacher online if you're threatening her because she gave you an F. But it would be OK to threaten her because she cut you off in traffic? That seems like a weird result. Would it matter if she cut you off in traffic on the way to school?

If a fellow student eggs your house, and you threaten him online, are you threatening him "because" of his status as a student? Well, on the one hand, no. He egged your house, and that's why you're threatening him. But on the other, he wouldn't have egged your house if you hadn't mouthed off to him in school the other day...

One could imagine that this provision is taken VERY literally, but then we get ridiculous situations where the only actionable threat looks like this:

"If you don't unenroll from school immediately, I'm going to box your ears in."

That's silly. No one makes threats like that. But I'm really at a loss to understand exactly what's supposed to be being required here.

So those are my thoughts on the specifics of the proposed law. I've got some more general concerns... like if this law is being passed because of concerns that off-campus online activity is currently protected by federal law, how is passing a state law going to change that, exactly? There's still this thing called the Supremacy Clause, and you can't take away federal protections through a state law. But I've gone on long enough for now.

I'll have some more abstract thoughts on this general topic in a day or two.