I loved the column "The Trouble with Rankings" by Carl Bialik at his Numbers Guy blog on the Wall Street Journal website. He gives valuable insights into the - occasionally very unscientific - way some of these rankings are obtained.
Here are a few highlights:
- Some magazines compute rankings in various categories and then add those rankings together, but if the range of results is very narrow, a city ranked 1st and a city ranked 100th might have much more similar profiles than the rankings suggest.
- Some magazines pick the wrong explanation for the data. For instance, "[t]he ranking [of American drunkest cities] counted arrests for driving under the influence against cities, though they could also reflect stepped-up enforcement of laws."
- US News hospital rankings use a reputation category that dwarfs all the other metrics, but "[h]ospitals with national reputations tend to win all available points in this category, giving them such a formidable lead that they typically top the rankings even if their scores on objective outcomes aren’t as strong... One result is that Boston’s Dana-Farber Cancer Institute ranked fifth for cancer treatment last year despite lagging behind many hospitals ranked below it on every measure besides reputation."
Bialik's print column offers an even more detailed and sobering account of the problems with rankings.
For more readings about numbers, another of Bialik's posts describes the difficulty in counting illegal immigrants in Arizona and the unverifiable assumptions that underlie those figures (the post is again a follow-up on one of his print columns, which has an interesting graph of the number of illegal immigrants as a percentage of state population per US state in 2008.)
In particular, studies "assume that about 10% of illegal immigrants aren’t counted in these surveys. But that figure largely is based on a 2001 survey of Mexican-born people living in Los Angeles," which - as its author emphasizes - should not be used to count immigrants outside LA County. Bialik also discusses the issues in using the number of border-patrol apprehensions as proxies.
The award for best quote of the day goes to the director of the Census Bureau, who stated: "We would like to do estimates that have the smallest number of assumptions we can’t test." A worthy goal indeed.