I read a very interesting article on bias. As we have discussed before bias permeates all forms of science and in particular forensic science. It provoked some thoughts. I wanted to pass it along in homes it might

The Trouble With “Limitations” In Science

By Neuroskeptic | May 16, 2013 4:09 pm

Is it always good thing to know your limitations?

Over at Scientific American, Samuel McNerney writes about the dangersof learning about common human cognitive biases. The problem is that it’s easy to find out about, say, confirmation bias, and think “Well, it affects other people, but now I know about it, I am immune to it” – and then proceed exactly as you did before, suffering the bias but now with misplaced confidence in your abilities.

I fear that a similar thing is at work in science, in the form of the Limitations Section.

It’s become fashionable for scientific papers to end with a few paragraphs about “the limitations of the work”. Why is this a problem?

Well, just as a morally confused Catholic might prefer to confess his sins regularly rather than change his sinful ways, the Limitations Section can serve as a kind of ritual cleansing which makes poor scientific practice seem acceptable. There’s a sense, usually unspoken, but I have heard it said, that“We acknowledge this limitation, so don’t criticize it.”

Worse still, if papers are getting published despite the fact that even the authors admit to a certain limitation, it would be easy to conclude that the limitation isn’t all that bad – so why should anyone bother fixing it?

Still, at least acknowledged limitations provide the reader with something to go on. Worse are the trivial ‘problems’ that often fill the Limitations Section, taking up room that ought to be spent addressing the real flaws. This kind of thing is actively misleading. To say that a handful of minor issues are the limitations is tantamount to saying there are no major ones.

When you think about it, it’s strange that scientists are expected to be their own critics in this way. Surely the flaws are for the reader to judge? Or the peer reviewers?

In fact, the reviewers are often the ones who write the Limitations Section, but I think this is often a way of passing the buck. When reviewing a manuscript, one sometimes feels that something about the study is flawed, not spectacularly so but seriously, and that in an ideal world it would prevent the paper from being published.

But rather than bite the bullet and reject the paper, you may feel – well, being realistic, everyone’s in the same boat, who hasn’t cut corners, and we’ve all got to earn a living… etc… – so you recommend the paper is accepted, but in order to avoid feeling like a walkover, you make the authors include your concerns as Limitations.

You act as the confessor for their scientific sins, in other words.