I found an interesting post about the Physics GRE exam on the Cosmic Variance blog of Discover Magazine, written by someone on the committee that prepares the questions. (The GRE [Graduate Record Examination] is an exam that students applying to graduate school have to take, a bit like the GMAT for business school and the LSAT for law school. The schools most often require the General Test, but some departments also ask candidates to take a Subject Test in one of eight fields: Biochemistry/Cell/Molecular Biology, Biology, Chemistry, Computer Science, Literature in English, Mathematics, Physics and Psychology.)
The post is intended for physics undergraduates who study for the GRE, but the comments it generated also raise interesting questions on whether a multi-answer quiz can correctly assess a candidate's understanding of physics. For instance, the author writes: "My first piece of advice to students studying for this exam is to focus
on reviewing the textbook from your freshman introductory physics
course. In my years on the GRE committee, when I have needed to consult
a text, it is that text at least 80% of the time. [...] I have found that only a small fraction of the items
on the GRE are actually from upper-level topics". My first reaction was: isn't it a little sad to test applicants' abilities based on a course they took as freshmen? At the same time, if they don't get the basics, it's unlikely that they will make good researchers.
The post's author explains that students only have an average of 1.7 minute to do each problem and that the GRE penalizes random guessing by substracting 1/4 times the number of incorrect answers, to the number of correct answers. He argues that "the Physics GRE really does test knowledge about basic physics and the ability to analyze physical situations accurately." But he does not comment on the GRE's usefulness in evaluating doctoral candidates beyond the fact that "[his] own observation is that students below about the 30% level have a very
hard time attaining a Ph.D., though this is by no means absolute." And if the GRE is 80% based on freshman physics, I can see why that would happen, and I would have quoted a threshold much higher than 30%. His other observation on the test's usefulness is that "[w]e do see a clear correlation between an incoming graduate student’s
Physics GRE score and their score on the other dreaded exam in a
physics student’s career, the Ph.D. written preliminary exam, which is
a very different beast." I guess that is better than no correlation at all with any milestone in a doctoral student's career.
Many commenters touched upon the issue of the test's usefulness. Commenter #4 writes: "It is astonishing and borderline disturbing that, after four years of
learning more and more about how to do physics, our academic futures
are contingent on recall of material from a class that we were
encouraged not to take." (The top students were apparently encouraged to take honor-level physics where they would focus on proofs and derivations instead of learning the formulas by rote.)
Commenter #5 also states: "[T]he emphasis on fast calculations in the physics GRE,
especially with numbers, though a reasonable skill for a physicist to
have, is not obviously more important in determining success than the
ability to formulate a mathematical problem from a set of physical
ideas. Nor is it even obviously related to the ability to solve a
particularly knotty differential equation. Nor is it obviously related
to the ability to design and execute a clever experiment." Also read Comments #6 and 7.
Here are some arguments in favor of the GRE: Commenter #9 points out that it is a lot more objective than GPA, which students can inflate by taking easy courses or by attending an university that sees a lot of grade inflation. (A 2001 Boston Globe story - Harvard's Quiet Secret: Rampant Grade Inflation, October 7, 2001 - reported that 91% of the students in the Harvard Class of 2001 graduated with some type of honors: cum laude, magna cum laude or summa cum laude, in contrast with 51% at Yale and 44% at Princeton. Also see this December 2001 article in the New York Times.)
The post's author, as Commenter #10, makes a similar argument, and points out that letters of recommendation vary wildly. I would add that creating a test which really allows students to demonstrate their understanding of physics would require a real person, as opposed to a computer, to grade the test, and raises issues of time and pay for these people. Commenter #61 points out that the Physics AP test "has both a multiple-choice section and a free-response problem section,
and many, many more high school students take that test than take the
physics GRE. So if the AP system manages to make it work on a much
larger scale, I don’t see why we couldn’t do the same thing as well."
Judging the ability to do research, which should be the purpose of the admission committee, is an impossible task in most circumstances, since graduate research work differs so much from undergraduate coursework and even undergraduate research projects (although a willingness to learn about research as an undergraduate should be a huge plus). Even graduate students who do very well in their first-year or second-year courses sometimes turn out to be below-average researchers, when they no longer have well-defined questions to answer. There might not be enough qualified students applying to doctoral programs; if there were, departments would not need to put together admits' weekends to entice admitted students to enroll. It does say a lot about the perceived lack of importance of doctoral-level work that high school students take a higher-quality test, in the sense that it better lets them demonstrate their understanding of the material they have learned, than doctoral applicants.