Math Moment

USA Today announced a few weeks ago that the College Board would stop "penalizing guesses" on AP tests. The formula used to compute scores had been: (number of correct answers) - (1/4)*(number of wrong answers), given that there were five possible answers for each question. I love the mathematical side of it. The idea behind the formula is that, if you have 100 questions and answer randomly each one, you have 1 chance out of 5 to find the correct answer although you do not understand the material. Counting only correct answers would yield an average score of 20, while in fact you really deserve 0. So your score is pulled back by 20 by dividing the total number of wrong answers (80) by 4, which is the number of wrong answers for each question.

Seen another way, your expected score for each question with 5 answers should be zero if you don't know the material, but the probability of guessing right is 1/5 and of guessing wrong is 4/5, so with w the weight for the number of wrong answers, you solve 0=(1/5)*1+(4/5)*(-w), which yields w=1/4.

Inside Higher Ed explains that the change in policy is due to the College Board's decision to redesign some of its AP courses, including the science ones. A draft of the proposed changes in the biology curriculum is available here (in pdf format); the goal is to increase depth of understanding by limiting the breadth of the content covered and using scientific topics drawn from cutting-edge research. The changes are classified into four areas: conceptual understanding, investigative skills, modern thinking in science and quantitative (computational) thinking. Specifically, "students will be encouraged to develop their ability to apply mathematics to wide sectors of biology so that they can better test hypotheses, model biological phenomena, interrogate complex data sets, and represent and interpret visualizations of relationships." I have never read the expression "interrogating data sets" before, but I love the idea. The ability to analyze data has certainly emerged as one of the key skills of scientists and engineers alike.

I couldn't figure out, though, why the change meant the end of the guessing penalty. Assuming multiple-answer questions are still offered (which is the only financially viable scheme to grade such a large number of exams in a small amount of time), students will always have the temptation to guess, so the only way the announcement makes sense will be if the deeper understanding they gain from the revised curriculum always allows them to eliminate some of the answers and make an educated (rather than uneducated) guess. It does seem a little premature to say that the changes will for sure lead to improvement in students' mastery of the concepts. One could argue that the whole concept of using tests with multiple-choice questions to assess deep understanding is flawed; I wrote about this here. But until computers can grade proofs correctly, those are the limits the College Board has to work with. 

The SAT, however, continues to penalize wrong answers.

Comments

[Ilya ^_^]

This is silly. Won't the result metrics simply be adjusted then to account for the score inflation?

[Aurelie]

Hi, Ilya - if you don't penalize for wrong answers, you can't distinguish the student who knew what he was doing but was slow from the one who guessed many questions and happened to get some answers right. But we have to see what kind of questions will be on the new AP tests to understand why the College Board decided to no longer penalize wrong answers.

[Paul A. Rubin]

I used the SAT method in an undergraduate course in organizational behavior (the significance of the topic being mainly that the audience was not composed of students for whom the term "expected value" has meaning). Despite my explaining early on that blind-guessing had an average return of zero but "educated" guessing (eliminating one or more clearly incorrect choices) had a positive return, a number of students apparently were intimidated into skipping questions about which they were uncertain (and then complaining bitterly on evaluation forms about the "unfairness" of the system). Perhaps the College Board decided that something in the planned revisions called for greater encouragement of (presumably "educated") guessing. Like Ilya, I assume that the conversion from raw scores to interpretable results will end up being adjusted to offset the removal of the penalty for incorrect responses.

[Aurelie]

Thanks a lot for your perspective, Paul! Sorry it took me so long to post this. For some reason I didn't see your comment until now. I thought your experience in your course very interesting.

You're right, I think we underestimate how obscure concepts such as expected value can appear to people who have received little to no quantitative training. I would much rather see a situation where people take computer-based tests where some fields are left blank for the student to fill. The computer would grade the test immediately and it would be much harder to guess right if the student doesn't understand the material.