Gyroscopic Investing

http://news.nationalpost.com/2014/11/12 ... mic-paper/

I take this to be an inevitable consequence of the fact that everyone is too busy to proofread papers.  That's because we're all regularly and increasingly deluged with requirements to fill out forms, take online courses, and formally ask permission for everything we do that is more complicated than visiting the rest room.

Somehow, no one seems to think that the ever-increasing regulatory burden is going to force attention away from things that have historically been done well...but of course it will.

Hilarious!:-D

I think another hidden message is published research are dog and pony shows.  None of them are trustworthy anymore.  I had to smirk at this comment:

But Carpenter suggested a more optimistic take: “Instead of a failure of the system, you could also consider this a success of post-publication peer review.”?

Oh, please!  The only success there is is about sizzle over steak or acting as an authoritative charlatan.

With all due respect....MG I really appreciate your posts, but on this one...no, peer reviewed literature are not "dog and pony shows".  You would be amazed at how much work goes into a published article, and how many eyes are on it.  That's why this was such a shocking event.

My publications take about a year from start to publication date.  On the way they generally get passed around our group of 5-10 people.  Then they go to the journal editor who decides whether to send it out for review.  It then goes to 2-4 people chosen by the editor.  They take a few weeks to review the article, and believe me they are MERCILESS...if I posted some of the comments I've gotten back on my papers I think you'd be shocked at the harshness.  Then I spend quite a lot of time responding to each point.  The paper makes the rounds of the group again, then goes back to review.  Sometimes it's accepted from there with additional revisions required by the editor, and sometimes it goes back for another round of reviews.  Of course, it can also be rejected in which case we start again at another journal.  After acceptance and return of the final manuscript, I then get the article proofs which have gone through a copy editor who generally has a list of questions for me to answer on structure etc.  Sometimes I catch errors at that stage, and once had to appeal to an editor to get them fixed.

Only then does the article actually appear in print.  By then of course I'm heartily sick of it!!

WiseOne wrote: With all due respect....MG I really appreciate your posts, but on this one...no, peer reviewed literature are not "dog and pony shows".  You would be amazed at how much work goes into a published article, and how many eyes are on it.  That's why this was such a shocking event.

Then how is so much flawed or crony literature published if theres so many eyes on it during the peer review processed?  Help me understand.

Like all human processes nothing's perfect...I just objected to the "dog and pony show" description. There are several reasons why flawed papers get through the system:

1. The paper is founded on commonly accepted dogma that is later proven to be erroneous.
2.  The reviewers don't have time to do a proper critiquing job, so they end up giving the paper an undeserved pass.
3.  There's a prominent name on the author list - or worse, said prominent person strong-arms the editor into accepting the paper.

What usually happens to these papers is that they get published, but eventually everyone realizes they're flawed and stops paying attention to them.

There's also a tiered system of journals.  Some journals, like Science and Nature, are very difficult to get into and it's very rare for one of those papers to be discredited.  Then there is the recent advent of online journals which are basically scams.  They spam advertisements and charge scientists a fee like $800 to submit papers - which of course only people who can't get their papers published in good journals will pay.  These undergo limited, if any, scrutiny.  Recently, someone I know submitted an article titled "Take me off your f-----g mailing list" and the text was that phrase repeated a thousand times.  It was accepted "pending payment".

You can get an idea of where journals are in the hierarchy by checking the impact factor.  Top tier journals have impact factors greater than 10, second tier are in the 8-10 range, and solid journals in the 2-8 range (roughly).  Also the authors' "H index" on their Google Scholar page can give an indication of how well respected they are.  Finally, it's helpful to see how many times a given paper has been cited.  This is pretty easy now since any paper's search engine entry will automatically include number of citations.

Hmm, good stuff, thanks!

I think the core problem, at least in the preventive health field, is once you apply the criteria you listed, you're not going to be left with enough surviving studies to definitely say one way or another that this is this or that is that.  But thank goodness for Examine.com!  Although very sparse right now in passable studies, they are upping everyone's game, but I think it will take some time before higher quality literature is published in response.  Competition is a wonderful thing (damn, did I just channel Buffett?) but it seems to happen verrry slowly in areas that desparately need it the most. 

I'm also encouraged by the NIH recently responding to grass-roots pressure and is going to soon require all clinical study results, both good and bad, be published publically.  Wow!  No more selective evidence to make the corporate interests look good while killing people at worst.  That's a pretty amazing middle-finger to Big Pharma!

WiseOne wrote: With all due respect....MG I really appreciate your posts, but on this one...no, peer reviewed literature are not "dog and pony shows".  You would be amazed at how much work goes into a published article, and how many eyes are on it.  That's why this was such a shocking event.

My publications take about a year from start to publication date.  On the way they generally get passed around our group of 5-10 people.  Then they go to the journal editor who decides whether to send it out for review.  It then goes to 2-4 people chosen by the editor.  They take a few weeks to review the article, and believe me they are MERCILESS...if I posted some of the comments I've gotten back on my papers I think you'd be shocked at the harshness.  Then I spend quite a lot of time responding to each point.  The paper makes the rounds of the group again, then goes back to review.  Sometimes it's accepted from there with additional revisions required by the editor, and sometimes it goes back for another round of reviews.  Of course, it can also be rejected in which case we start again at another journal.  After acceptance and return of the final manuscript, I then get the article proofs which have gone through a copy editor who generally has a list of questions for me to answer on structure etc.  Sometimes I catch errors at that stage, and once had to appeal to an editor to get them fixed.

Only then does the article actually appear in print.  By then of course I'm heartily sick of it!!

I'd totally back up what WiseOne says but also add that peer reviewers often also ask for extra experiments to be done or to be redone in a different way. When it is working at its best, peer review can help an inconclusive investigation to become a solid advance. But as WiseOne says, everyone involved is human and people can and do act stupidly and so sometimes good work gets thwarted by reviewers asking for time wasting additional work or bad work doesn't get spotted. For what its worth, I've done my damndest to ensure that papers I've authored or reviewed aren't misinformation and all of the peer reviews I've been subjected to have come across as conscientious efforts to ensure the science was as solid as it could be.

I'd also like to add that what gives me confidence in science is when subsequent work gets done that is based on the findings of previous published work. If things turn out as predicted by the previous work THEN that previous work really becomes part of the knowledge foundation. Sometimes things don't turn out as predicted and it becomes necessary to unpick how things were misinterpreted -that's the process.

stone wrote: I'd totally back up what WiseOne says but also add that peer reviewers often also ask for extra experiments to be done or to be redone in a different way. When it is working at its best, peer review can help an inconclusive investigation to become a solid advance. But as WiseOne says, everyone involved is human and people can and do act stupidly and so sometimes good work gets thwarted by reviewers asking for time wasting additional work or bad work doesn't get spotted. For what its worth, I've done my damndest to ensure that papers I've authored or reviewed aren't misinformation and all of the peer reviews I've been subjected to have come across as conscientious efforts to ensure the science was as solid as it could be.

I'd also like to add that what gives me confidence in science is when subsequent work gets done that is based on the findings of previous published work. If things turn out as predicted by the previous work THEN that previous work really becomes part of the knowledge foundation. Sometimes things don't turn out as predicted and it becomes necessary to unpick how things were misinterpreted -that's the process.

+1!!!  And yes, I too have worked hard on every grant and paper to make sure it's solid.  But anyone can make mistakes and sometimes the process of acquiring new knowledge is bumpy, as stone described.

I doubt any source (or website) can claim to have superior knowledge of the literature.  God knows I don't and there are a lot of extremely smart people in the scientific community who all know that they can be fooled just like anyone else.  I think the best advice I can give if you're researching a specific topic is to read several papers and also papers that cite the ones you're looking at, be aware of the sources of information, and acquire a good "B-S" detector.