I recently saw all 10 episodes of the PBS documentary The Vietnam War by Ken Burns and Lynn Novick, a superb documentary for the most part, and for today's post I wanted to talk about Robert McNamara's misguided emphasis on quantitative metrics to determine whether the war was being won. In particular, McNamara focuses on a single metric, the kill ratio (how many U.S. troops killed vs how many North Vietnamese troops). This led U.S. troops to classify civilian victims as enemy combatants and inflate their numbers, giving the impression the war was being won. So the first issue was the definition of measures, which led to distortion in their computation.
Also, in the documentary someone makes the excellent comment that it didn't matter to the American public if the kill ratio was 10:1 (10 North Vietnamese soldiers killed for each American soldier), because all that Americans cared about was that one U.S. soldier who got killed. At issue then was also the interpretation of measures.
In academia it is easy to come up with examples of flawed metrics. For instance, using only GPA to assess a student's performance in college may result in grade inflation if an instructor doesn't want to deal with students arguing for a better grade. Ranking students, maybe not with a precise rank to avoid cutthroat competition but in terms of percentile, as is done at the GRE exam (and in college entrance exams in France and Turkey, for instance), could provide good additional information. That is also a purpose that graduation "summa cum laude", "magna cum laude" and "cum laude" could serve, although the requirements for each vary from school to school.
Relying on teaching evaluations to assess teaching can encourage being overly lenient so that students will give good grades to the instructor. I remember in high school in the French system our teachers were assessed through a visit from an official from what would be a cross between the U.S. school district and the state. (These visits didn't happen very often but it was a big deal when they did.) Maybe having peer evaluations of one's teaching in addition to teaching evaluations would help gain a more accurate picture of someone's teaching ability.
Of course, there is also the issue of quantifying research output through a researcher's number of publications. A recent article in the New York Times discussed the situation in China, which "has retracted [since 2012] more scientific papers because of faked peer reviews than all other countries and territories put together." And new journals keep proliferating to serve as outlets for more academics' work. Then you need to rank those journals, typically through a metric called impact factor, and then journal editors try to boost their impact factor by asking authors to cite other papers published in the journal or commissioning review papers. Authors themselves may sometimes ask authors of papers they are refereeing anonymously to add references about their papers. (Those cases are relatively easy to spot, either there is only one reference with only tenuous link to the paper being reviewed, or there are several references to be added that have one author in common.)
If you give people a metric, they will try to use it to their advantage. In the case of the Vietnam War, the consequences were disastrous.
The army has come a long way since those days and tried to learn from its mistakes, as shown in this RAND report on assessment and metrics in counterinsurgency. Its author argues that centralized assessment fails in counterinsurgency, and should be replaced by a bottom-up contextual assessment process. I've been thinking about how this can translate into evaluating universities. I guess the equivalent would be that each department has to evaluate itself and then pass along its assessment up in the hierarchy. But how would the results be validated and communicated to the outside public? How to incentivize being truthful in the never-ending quest for resources? Wars have a clear result at the end. It is much harder to tell whether a university was successful in rising above another one. Yet, we all continue to keep an eye on the rankings of U.S. News & World Report while pretending we don't care.
My other thought was: what would (above-board, PG-rated) guerrilla warfare look like in the context of universities vying for better national spots? Would that relate to Clayton Christensen's disruptive innovation, with the up-and-coming university staying under the radar until it has siphoned many potential applicants, whether faculty or student, from the incumbent?