Science Isn't Dead, But it Is Ailing! [Jay Guevara]

Science faces a crisis of confidence owing to its politicization and the replication crisis. The latter, which intersects with concerns about scientific fraud, is the subject of this post, a matter on which I've previously commented. (OK, ranted.)

Part of the impetus was from a study by Amgen, summarized in Wikipedia as follows:
Out of 49 medical studies from 1990–2003 with more than 1000 citations, 45 claimed that the studied therapy was effective. Out of these studies, 16% were contradicted by subsequent studies, 16% had found stronger effects than did subsequent studies, 44% were replicated, and 24% remained largely unchallenged.[59] The US Food and Drug Administration in 1977–1990 found flaws in 10–20% of medical studies.[60] In a paper published in 2012, Glenn Begley, a biotech consultant working at Amgen, and Lee Ellis, at the University of Texas, argued that only 11% of the pre-clinical cancer studies could be replicated.[61][62]

To try to put some numbers to my visceral impressions, and thereby to test my conjectures regarding dodgy results I went over to Retraction Watch, which features a database of papers that have been retracted by the authors/journals or otherwise had their validity called into question.

One can rummage around in that database for hours, so I narrowed the search to "concerns/issues about results," which yielded 562 hits.

Where, geographically, do these problems come from? Querying the database for papers that raised concerns/issues with results (regardless of whether from physical, life, or medical sciences) yielded these results, again not normalized for the number of papers published in each country:

Interestingly, the database includes data regarding some of the more dubious areas of "science." For this search I included outright scientific fraud, and found effectively none for any of these fields, although a similar search the physical, biological, and health sciences all returned a number of hits. Once again, the caveat regarding total number of papers published applies, this time probably a forteriori.

For comparison:

My conclusion, subject to the caveat above: the biomedical literature does in fact have grievous problems, and China is punching above its weight in generating lousy results. Last, the Wild West of "scientific research" (climate change, sociology, alternative medicine, and psychology) I suspect may not have fraudulent papers because results in those areas are so squishy that it is impossible to falsify pretty much any assertion made in those literatures.

posted by Open Blogger at 07:30 PM

| Access Comments