An article in Scientific American on April 21 revealed an insidious trend in the world of basic science — selectivity for positive results.
The article focuses in particular on studies of mindfulness, a therapeutic method based on judgment-free reflection upon one’s own thoughts and feelings.
Mindfulness studies were the subject of a recent review, which noted how highly unlikely it is that negative results related to mindfulness as a mental health treatment are being reported as often as they should.
Positive findings surrounding mindfulness are reported about 60 percent more often than is likely to happen, according to the review.
This alone is enough to raise concern, but taken in combination with a 2014 review across disciplines which showed only about 36 percent of clinical trials at major research centers are reported within two years of trial completion, it indicates a serious deficit in potentially important negative information.
The first study indicates the ability for publication bias to cast a treatment in a light that may be falsely positive, while the second shows this could conceivably happen anywhere and relate to any type of study.
In the best-case scenario, the pressure to publish positive findings leads us to ignore certain irrelevant information, dismissing it as unnecessary and allowing us to move forward with more important investigations.
Unfortunately the results of this study indicate that we probably don’t occupy the best-case scenario.
We’ve gone beyond the point of simply ignoring things that are uninteresting or not useful, so much that we now ignore information that can potentially let us know that our time and money would be better spent on something else.
In short, our negative findings are being heavily undervalued.
Therein lies the real problem — publication bias can make a particular treatment seem very promising, when in fact it holds very little potential benefit.
A recent example of this is noted in the article — the popular notion of ego depletion failed to hold up in a recent replication trial. It is suggested that this may be the result of skewed data benefiting the idea’s image.
A number of countermeasures have been suggested for this bias, including the #bringoutyernulls tag on Twitter, which began to gain a foothold circulating in academic communities.
People have tweeted findings they found weren’t significantly related to the hashtag, with the intention that doing so would shed some much-needed light on negative findings within the scientific community.
A method proposed by the Scientific American article, one with a bit more bureaucratic rigor, is that of pre-registering trials for publication.
This would allow journals to select topics they’d like to see articles on, register a paper on that topic before the study is complete and then be bound to publish that study regardless of whether the finding comes back null.
This would help with publication bias because it would make journals more inclined to include content on important topics, rather than only rewarding positive findings with publication.
