Peer Review Is an Awful System, but It's Better Than All of the Other Awful Systems: The McKinsey ACA Report Edition

Peer review, the process in which scientific publication has to pass muster with critical reviewers, has a lot of problems. But a widely-cited report by McKinsey & Company consulting which claimed that Obamacare will result in one-third of companies dropping healthcare coverage for their employees is worth examining (italics mine):

The predictable fallout led Democrats, and several reporters, to press McKinsey for the survey itself -- a request McKinsey has declined on the grounds that the material is proprietary.

But multiple sources both within and outside the firm tell TPM the survey was not conducted using McKinsey's typical, meticulous methodology. Indeed, the article the firm published was not intended to give the subject matter the same authoritative treatment as more thorough studies on the same topic -- particularly those conducted by numerous think tanks, and the Congressional Budget Office, which came to the opposite conclusion. And that's created a clamor within the firm at high levels to set the record straight.

"This particular survey wasn't designed in away that would allow it to be peer review published or cited academically," said one source familiar with the controversy....

What was one of the problems? Well:

Lastly, this tidbit was included in the McKinsey Quarterly article about the survey:

"...our survey educated respondents about [employer sponsored insurance] implications for their companies and employees before they were asked about post-2014 strategies."

In other words, those conducting the survey may have primed respondents to say they would keep or drop coverage.

And there's no potential conflict of interest either (italics mine):

A third source speculates that the firm may have reached its outlying conclusion by basing its questions on the firm's own advice to clients on how best to arbitrage the new reforms. Specifically, under the law, employers could devise their benefits packages in ways that makes them unappealing to lower-income employees, who would then have to enter the exchanges. Though TPM could not confirm this, the conclusion is supported by a disclosure within the McKinsey study itself.

There are all sorts of problems with federally-funded research and peer review. But this scandal--and it is a scandal, although no penises were photographed--should put things in proper perspective.

More like this

You don't have to look far to find mutterings about the peer review system, especially about the ways in which anonymous reviewers might hold up your paper or harm your career. On the other hand, there are plenty of champions of the status quo who argue that anonymous peer review is the essential…
Yesterday was the end of the first open enrollment period for people buying private health insurance plans on the federal and state-run health insurance exchanges. President Obama announced today that more than seven million people enrolled in private plans, helped by a surge of signups in the few…
As if outing Valerie Plame, whose primary task was to monitor and contain WMD proliferation in the Middle East--including Iran, wasn't bad enough, the Bush Administration destroyed another intelligence gathering operation for political gain (italics mine): A small private intelligence company that…
A while ago, I described how previous decisions allowing the patenting of human genes--and thereby making cheap, affordable diagnostics impossible--flew in the face of the goal of federally-funded research. There's no reason to patent something and charge thousands of dollars for something a high…