Time to end the data massage

Matthew Partridge has a gentle reminder for us all: experiments don’t lie; researchers lie when they don’t like the experiment and ‘correct’ the data… 

Scientific progress is a tricky thing. Despite what you might think, the direction of science is not always forwards – sometimes as a species we can unlearn things which then take us hundreds of years to re-discover. Sometimes this is simply from ideas not being publicised enough and slipping through the cracks. But more often, it’s from someone disagreeing with the outcome and wilfully deciding to ignore or deflect it.

But on the smaller scale, there are cases where scientists set back science without even realising they were doing it

Now there are different scales to this. On the one hand you can have, say, those holding international geopolitical positions that suggest CO₂ isn’t a greenhouse gas – which I think is akin to looking at a bear covered in blood and surrounded by shredded tents and saying: “The campers probably just tripped and got horribly mutilated on their own”. But on the smaller scale, there are cases where scientists set back science without even realising they were doing it. In science, results can do three things. They can teach you something, they can teach you nothing (rare) and they can mislead you.

Experiments don’t lie; researchers lie when they don’t like the experiment and ‘correct’ or crop the data

Results that teach you something don’t have to be good results. Failed experiments can teach you plenty – even if it is how not to do something. Results that teach you nothing are often results that are repeats of experiments you might be doing as due diligence. Results that mislead you are, in fact, lies. Experiments don’t lie; researchers lie when they don’t like the experiment and ‘correct’ or crop the data.

It can start off innocently enough. Even minor ‘fudging’ of data points, which can seem like the right thing at the time, can obscure deeper information. It can take many forms – from deleting outliers without taking the time to consider why they are there, to removing a dataset because the others don’t agree with it, right up to the wholesale invention of data because you didn’t have time to run a repeat. Whatever the reason, instead of showing actual data doing actual data things (like be weird and unreliable), you are showing made up data.

I often hear people say that it’s a known problem – a background state which really “is not helping science progress”. I very strongly disagree – massaging data doesn’t leave scientific progress stationary, it actively sends it backwards. As a researcher I want to make progress. Not every experiment I do progresses either knowledge at large or even my own all that much. But I can say that my experiments don’t actively make science go backwards.

To me, that is like being a lifeguard who is not only bad at saving people but actively drowns the occasional slow swimmer just to make sure it looks like he’s doing something...

Dr Matthew Partridge