Most scientists can't replicate studies by their peers

[Kriegsspiel]

I predict John Michael Greer is going to have a field day with this.

[I Shrugged]

self-deleted, not relevant.

[Maddy]

I'm a little skeptical of the premise. If this were occurring on a large scale, wouldn't the scientific community be the first to know? Mainstream researchers don't work in isolation; they work in academic departments surrounded by co-investigators and peers. I cannot imagine how an investigator whose work is not reproducible would last long in that environment.

From time to time, you hear of a researcher who's dummied up data--ostensibly for the purpose of padding his or her CV, but from what I've heard, that's a very big deal--not to mention a guaranteed career-killer.

So is the "science" that's being discussed here mainstream basic science, or is it "social" science--or possibly even politics masquerading as science? "Studies" such as the infamous Hyte Report, which is now largely recognized as having been politically motivated and impossibly biased, comes to mind.

[Maddy]

That quote is as clear as mud. According to the article, it's not about investigator fraud, but rather a "culture" that promotes "flashy findings" and "exciting breakthroughs," and that measures success based upon the number of publications with the researcher's name on them. It's a problem of the "system" which, the author claims, "encourages less than optimal outcomes." So nobody's at fault, and everybody's to blame. How progressive.

What is this author really saying? That there's not only widespread fraud, but that the entire university system is in on it?

Maybe it's the field of journalism that could stand a little firming up.

[Maddy]

Where's WiseOne?

[Pointedstick]

I believe it. I encountered it myself, in fact. About a year ago I got really into geopolymers and read close to 100 scientific journal articles recording examples of the materials being synthesized. I followed the experiments outlined in the articles with materials and procedures that were as close as possible to the ones they used. I was often able to replicate the results--but only often, not always. In fact, I discovered that a large number of the articles described procedures that were clearly incorrect, or even impossible. Some omitted key pieces of information, making the experiments un-replicable. Some of the papers even confused two related concepts, and did not actually create geopolymers--instead simply creating simpler alkali-activated cements.

It was an eye-opener. I can only imagine it's even worse in the social sciences where much of the research is inherently un-replicable.

[boglerdude]

Worked in a lab for a bit, and it's bad. The incentives are to make shocking discoveries, not to publish boring but useful negative results. And refuting another lab's work is awkward. Whereas if you confirm another lab's bad data, and not many labs work in that niche....thats easy money for both labs.

IMO science needs to get boring and not fund exotic ideas. Do conservative and high output industrialized things like the Human genome project.

[FarmerD]

“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.” Dr. Marcia Angell.
https://www.amazon.com/Doctoring-Data-m ... m+kendrick

[WiseOne]

Just had to weigh in on this one. Not only am I one of the scientists in question, but I'm currently traveling to take part in a federal review panel, for evaluating the scientific merit of submitted grants applications. So quite a timely topic, since this is very much part of the review process.

I am one of the few people who like to read the methods section in published articles - and indeed, I often find them to be a mess. Parts are often lifted from other papers, there's contradictory points and critical info left out. If I really want to know how something is done though, I just drop an email to the corresponding author and ask. That's quite common.

There problems I see that are more fundamental than this: hidden assumptions that are just plain wrong. An example is that most statistical tests assume a Gaussian (normal) distribution of the data. In biological science, this is very often not the case. I usually run bootstrap tests to determine significance thresholds, or stick with non-parametric tests. The tons of papers out there on fMRI or (worse) resting state fMRI (which a friend of mine humorously calls "sex-state fMRI" since we all know what people think about by default) are mostly invalid because of this. There was a great article recently in PNAS estimating that about 70% of fMRI studies are plagued by false positive results.

Multiple comparisons are another problem: if you run thousands of tests each with a false positive probability of 0.05%, you are almost guaranteed to end up with a false positive result. This is my big problem with the entire field of genetics: it's based on this kind of multiple comparison test, with thousands of genetic markers tested independently. Genetics researchers know about this problem, but they have yet to do anything to address it. They don't want to, as it would threaten the mill of easy papers that current methods produce.

Clinical research is worse yet. It is astounding how many people don't understand that correlation does not necessarily imply causality. You can show an association between high blood pressure and heart attacks until you are blue in the face, but you will not prove that lowering blood pressure reduces heart attack risk until you test specifically for that with a controlled trial. Then we come to the dark side of clinical trials for new drugs or devices: subtly manipulated statistical methods to produce desired results, in a setting where there is a LOT of money at stake. For example, I had reason recently to comb the literature looking for points of comparison between a new and much ballyhooed device, and an old and non-sexy device with the same indications. A different measure was used to assess the new device that was chosen specifically to make the results look as good as those reported in the trials for the old device - but once you realize this, you realize that the new device is not as effective as the old one, despite being more invasive.

All this sounds really discouraging and it is, but there is plenty of good science that occurs in spite of all this. Be most suspicious of clinical (associative) research and "hot topics" like genetics and fMRI, which tend to attract sloppy thinkers. I went for a non-hot topic field and ended up doing well and getting plenty of recognition, and I'm glad I stuck to my guns

[Maddy]

Yes, very interesting--but troubling. WiseOne, putting aside the issue of methodologic and statistical errors (which would seem to affect the validity of the results but not their reliability), what is driving this trend? Is it outright fraud? What is the response of other scientists, and of the academic institutions in which they work, when an investigator's work cannot be reproduced?

For a time, I worked in a medical research lab back in college, but that was 40 years ago. As I recall, there was a great deal of collaboration among researchers and among institutions internationally--not only with respect to particular studies, but to particular lines of study. One guy's research was often built upon the foundation of some other guy's research. There was a lot of exchange around ideas and techniques. It would have been difficult, if not impossible, to escape notice if methods were sloppy or results suspicious. Has something changed since then, or was I wearing rose-colored glasses?

[Pointedstick]

Yes, very interesting--but troubling. WiseOne, putting aside the issue of methodologic and statistical errors (which would seem to affect the validity of the results but not their reliability), what is driving this trend? Is it outright fraud? What is the response of other scientists, and of the academic institutions in which they work, when an investigator's work cannot be reproduced?

For a time, I worked in a medical research lab back in college, but that was 40 years ago. As I recall, there was a great deal of collaboration among researchers and among institutions internationally--not only with respect to particular studies, but to particular lines of study. One guy's research was often built upon the foundation of some other guy's research. There was a lot of exchange around ideas and techniques. It would have been difficult, if not impossible, to escape notice if methods were sloppy or results suspicious. Has something changed since then, or was I wearing rose-colored glasses?

I was curious about this myself, and when I kept reading chemistry papers that exhibited this problem, I did some sleuthing. The papers with the most problems were either not cited at all by any other papers, or else cited only as supporting evidence. Never were their experiments actually replicated.

There are so many more papers published today compared to 40 years ago that perhaps a lot of the bad ones are simply getting lost in the noise, because they aren't interesting enough or written by prestigious enough figures to be cited or replicated.

I also noticed that the shoddiest papers invariably came from outside the USA. Southeast Asian countries popped up the most often.

[Mountaineer]

Great post, WiseOne. Thanks.

Ditto! Thank you.

[Maddy]

Another thing that may have changed over the last 40 years: I have a very vivid recollection of research as being an incredibly slow and tedious process. I'm the kind of person who needs a fairly constant dose of reinforcement in the form of tangible successes, and judged by that standard, life in the laboratory was incredibly boring. "That's the nature of basic research," I was told. "You make tiny incremental advances which, standing alone, don't mean a whole lot. Maybe you'll stumble on one really significant thing in the course of your career, but you'll be lucky if anybody other than those in your immediate professional peer group even understands it."

So with that in mind, I was struck by this article's reference to an academic environment propelled by "flashy findings" and "exciting breakthroughs." Maybe that's something that's changed, also.

[dualstow]

I also noticed that the shoddiest papers invariably came from outside the USA. Southeast Asian countries popped up the most often.

I wonder if that Korean guy finally has my clone ready. Ordered it years ago.

[WiseOne]

Yes, very interesting--but troubling. WiseOne, putting aside the issue of methodologic and statistical errors (which would seem to affect the validity of the results but not their reliability), what is driving this trend? Is it outright fraud? What is the response of other scientists, and of the academic institutions in which they work, when an investigator's work cannot be reproduced?

For a time, I worked in a medical research lab back in college, but that was 40 years ago. As I recall, there was a great deal of collaboration among researchers and among institutions internationally--not only with respect to particular studies, but to particular lines of study. One guy's research was often built upon the foundation of some other guy's research. There was a lot of exchange around ideas and techniques. It would have been difficult, if not impossible, to escape notice if methods were sloppy or results suspicious. Has something changed since then, or was I wearing rose-colored glasses?

It's troubling indeed. Lots of reasons for this. To list a few:

Grant funding is tight and at the same time institutions are less forgiving of funding gaps so people are getting desperate. Everyone is writing grants and papers fast to keep their heads above water. Quality gets sacrificed in the process.

For similar reasons people are running to the hot topics, which waters down the science further.

there is lots of cruddy work from places like Mexico and China, but they don't make it into the good journals. Usually.

In the long run though, a papers true worth is recognized. Citation indices are a decent measure of the value of a given article. Science is still slow and plodding, and cutting corners doesn't really work as a long term strategy.