Fake it ‘til you make it…

In the hunt for dark energy, cosmologist Dr Massimo Viola has become embroiled in a tale of deception most foul…all in the name of the truth. Here he tells us why the thrill of fake data may well be integral to beating confirmation bias in science  

In 2011 three scientists won the Nobel Prize in Physics for discovering that the expansion of the Universe is accelerating¹. The discovery dates back to 1998 and it came as a complete surprise to the scientific community. We knew back then that the Universe was not only made from ordinary matter, but also a mysterious form of dark matter which doesn’t emit light but does regulate the motion of the visible stuff. The expectation was that all this matter should have made the expansion of the Universe slow down, but instead the Universe is accelerating. To date nobody knows why.

Yes, it is possible that after all, the acceleration ‘theory’ of the Universe is just a result of our calculations, based on Einstein’s theory, being wrong

A lot of different experiments started after this amazing discovery with the aim of measuring the expansion history of the Universe more precisely. At the same time scientists started developing a plethora of new theories to explain the nature of this new phenomenon. Is the expansion of the Universe really caused by a new force – ‘dark energy’ – or is Einstein’s general relativity wrong? Yes, it is possible that after all, the acceleration ‘theory’ of the Universe is just a result of our calculations, based on Einstein’s theory, being wrong.

Cosmological experiments aiming at the investigation of the dark Universe are very complex. Not a surprise, as we try to understand how the Universe works from measuring the light, shape and the position of objects which can be billions of light years away from us. We constantly make choices on how best to make measurements – so in order to extract meaningful information from a huge amount of data, we write a lot of complex software. But when do we stop questioning the data and the code of this software? This was exactly the question we asked ourselves when we started working on a very ambitious project, the Kilo-Degree Survey (KiDS)².

The goal of this project was to map the dark matter distribution at different cosmic epochs, which is one of the most direct ways to precisely measure the acceleration of the Universe and study this mysterious dark energy. We made use of the fact that light from distant objects is bent by intervening (dark) matter. This phenomenon, called gravitational lensing, is one of the most powerful techniques to probe the dark Universe. In practice we measured how much the shape of distant galaxies was modified by this lensing effect and this allowed us to reconstruct the dark matter distribution through which the light propagated.

When we started the KiDS project, two other experiments had just published their results. The Planck collaboration published the tightest constraints on cosmological parameters to date based on measurement of the cosmic microwaves background radiation³, while the CFHTLenS collaboration published the tightest constraints on cosmology using the weak gravitational lensing effect⁴. The big surprise was that the two results disagreed. Did one of the two teams do something wrong in the analysis? Or was that a hint that something was wrong in the model we used to interpret the measurements? That would be truly exciting! My colleagues and I were very clear from the beginning of the KiDS project that, as much as possible, we would try not to be influenced by previous results. We wanted to avoid something known as confirmation bias: When you are expecting a particular result and find it, you tend not to question your data any further.

Working with fake data is common practice in particle physics experiments and in the gravitational waves community

So how do you prevent this confirmation bias from happening? The answer is fake data. Working with fake data is common practice in particle physics experiments and in the gravitational waves community. We are one of the first teams to embrace this method in studies of the dark Universe. At a very early stage, right after our pipeline delivered the first catalogue, we asked a colleague – external to our collaboration – to mess with the data. He produced two copies of the original catalogue, slightly perturbing the measurements, and then gave these back to us with the original catalogue, without telling us which one was which. From that moment on, we were blinded to the truth – next all the analysis needed to be repeated thrice, and of course each catalogue gave a different answer in the end.

It was a thrilling experience. From the moment we got the three catalogues to the time we stopped questioning the data and we got the final results, almost ten months passed. The results from catalogue number one agreed with the measurement of the Planck collaboration, the results from catalogue two were very much in tension with those results and catalogue three agreed well with the constraints published by the CFHTLenS collaboration.

We wrote three versions of the paper we were about to publish and when the whole team was happy with the analyses, we went back to the colleague who had created the fake data and asked them to reveal which of the three catalogues was the true one. By that time, there was open betting in the team about which one would be the true catalogue! The rules we set ourselves were quite strict: no matter which catalogue was the true one, we were not allowed to change any aspect of the analysis unless we documented it in the paper. The end of the story is now published in Monthly Notices of the Royal Astronomical Society⁵. The true catalogue ended up being the one giving results in agreement with the previous finding by the CFHTLenS collaboration.

The discrepancy with the Planck results remains a mystery and several teams are currently investigating it...obviously blinded.

Author: Dr Massimo Viola is a post-doctoral fellow at the Leiden Observatory in the Netherlands. He focuses on better understanding dark matter, dark energy and galaxy formation.

References 1 https://www.nobelprize.org/nobel_prizes/physics/laureates/2011/press.html 2 http://kids.strw.leidenuniv.nl/index.php 3 https://arxiv.org/abs/1502.01589 4 https://arxiv.org/abs/1303.1808 5 https://arxiv.org/abs/1606.05338