Community Blogs

Statistical Significance

Ok, so I’m someone who gets jaded really easily, keep that in mind while reading the following.

I’ve done two summer studentships, have just about finished my B.Med.Sc (Hons) thesis, and have earned some $$ doing data collection for nation-wide studies. So I’m hardly a seasoned scientist, but I guess for an undergrad-entry medical student I’m doing alright on the science front.

And wow.

I’m loving research. It lets me use big words, and I can now nod at the correct times when smart people speak to me. And lets face it, also nicely strokes my inflated-but-fragile ego. Aside from that, it’s really awesome getting to work with people who are internationally up there in their fields, and figuring out how the whole ‘evidence-based medicine’ wheel actually goes around.

But that’s where I wince a little.

‘Evidence-based medicine’. In medical school, and on popular social media accounts like the ‘I f**king love science’ facebook page, it is seen as some peer-reviewed Ivory Tower from which the bold knights of science ride forth. So long as we have the peer review process and that all important ‘statistically significant’ p < 0.05, our evidence is infallible, and we can march forward clinically with the sound knowledge that what we are doing is based in fact.

Except for the part where an awful lot of it is bullsh*t. Let me explain:

– Peer review:

Oh peer-review. It’s almost comical how this term is used as though it means that a group of impartial experts in the field in question have sat down and gone through the study from start to finish with a fine-toothed comb. Getting my study ‘peer-reviewed’ for the sake of ethical approval essentially meant sending it to my supervisor’s friend to be checked for grammatical errors. And, perhaps naively, I had always assumed that when papers were peer-reviewed the raw data went with them. Lol. The ‘raw data’ is just an excel spreadsheet or six you’ve entered all your data into, nobody wants to see those. And, as I’ve come to realise, anything can be ‘peer-reviewed’, the words almost become dangerous when the likes of homeopaths ‘peer-review’ each other’s drivel.

– What you’d ideally do vs. what you actually do:

Pure science can only really happen in 100% controlled labs, and even then that doesn’t happen all the time. If you’re doing clinical research, you’re going to have *so*, SO many times where your lovely-looking protocol has to bend to allow for the realities of working with other human beings, in a clinical environment.

– Pure errors:

The tired B.Med.Sc (Hons) student, entering data into different statistical programmes like a zombie. Plenty of it can’t be copy and pasted. So tonight I found myself wondering what proportion of the data I’ve entered has mistakes in it. *shudders*.
Let me walk you through all the places where people f**k up, and the numbers change. Firstly, recording things. Chances of you repeating each measure perfectly for each subject? Nil. So your data is wrong from the get-go. And then you have to write it down. Chances are, if you’re recording 50-100 things, you’ll have written the wrong numbers for a few of them. Then you have to enter it into your computer. All it takes is a few incorrect key-strokes and boom, your dataset has been slightly altered. Then there’s the part I mentioned before, where you have to move it around and make it jump through some hoops. More mistakes. Then you’ve got to interpret it, and when there’s a string of a million blimmin’ numbers in front of you, you’re going to do things like look a the wrong ones. Then you have to get it into a report. Then the way you describe your data has to accurately represent it, which brings me to….

– Bias:

Whether it’s because one participant was really nice, so you took extra care doing their measurements, or because you think you’ve found a cool association and start hunting for ways you can analyse the data to prove it, biases are there. I’m not sure what’s scarier – the fact that you don’t even know what a lot of them are (and therefore can’t control for them), or the fact that a lot of them you’re sort of semi-consciously aware of, but ignore. Bias could come from things you have little to no control over, such as the nurses only sending through to you the participants they think can cope with your test. Or it could be something you sort-of know is happening, like your mood if it’s been a terrible day, making your judgement calls in data analysis different that day than the previous one.

– Statistical significance:

So there’s a few problems with this. 1) Having to achieve 95% (or in some cases even 99%) confidence means that a lot of true associations get missed simply because the sample size was too small for the effect size. 2) The flip-side of this, is that it’s 95% confidence. Which means that one in twenty conclusions drawn that are ‘statistically significant’ are wrong, even if the science was done perfectly. 3) ‘Statistical’ significance often means nothing. It’s kind of frustrating that they picked the word ‘significance’, which carries connotations of importance, even if all it is is a statistical association. Which leads me to…

– Reporting:

Step 1) Oh look, this thing killed cancer cells in a Petri dish slightly better than this other thing (p = 0.049).
Step 2) I kind of need to get more grant money, so if I could publish this in Nature it’d be great. I’ll give it the title ‘Substance X – Towards a cure?’.
Step 3) My university’s press officer will get hold of this, and tell the media there’s something big happening.
Step 4) The media then release it on nation-wide news as a scientifically-proven cure for every cancer ever.
Step 5) People start badgering their oncologists for it.

Need I say more?

Ok, so I’ll stop there, although I could go on for a while. Let’s not forget that, despite the fact it’s quite clearly nearly impossible to actually get bullet-proof evidence, particularly for a lot of things in medicine which you can’t ethically run an RCT on (e.g.: defibrillators vs. placebo), we have to at least try. The fact that we actually try to improve the safety and efficacy of our practice is what sets us apart from the homoeopaths and the chiropractors.