Reproducibility with VIP – ReproVIP

In the last few years, there has been a growing awareness of reproducibility concerns in many areas of science. In a recent study, the analysis of a single neuroimaging dataset by 70 independent analysis teams reveals substantial variability in reported results, with high levels of disagreement across teams of their outcomes on a majority of tested pre-defined hypotheses. Despite the increase in awareness and a growing number of projects tackling this lack of reproducibility and proposing various tools to improve it, researchers still lack an integrated, end to end solution, providing a good level of reproducibility at a reasonable effort.

In this context, ReproVIP aims at evaluating and improving the reproducibility of scientific results obtained with the Virtual Imaging Platform (VIP) in the field of medical imaging. We will focus on a reproducibility level ensuring that the code produces the same result when executed with the same set of inputs and that an investigator is able to reobtain the published results. We will investigate reproducibility at three levels: (L1) the code itself, and in particular different versions of the same code, (L2) the execution environment, such as the operating system and code dependencies, parallel executions and the use of distributed infrastructures and (L3) the exploration process, from the beginning of the study and until the final published results.

At Creatis, since 2011, we have developed and deployed VIP, a web portal for medical simulation and image data analysis. By effectively leveraging the computing and storage resources of the EGI federation,VIP offers its users high-level services enabling them to easily execute medical imaging applications on a large scale computing infrastructure. In 2021, VIP counts more than 1200 registered users and about 20applications. In the last few years, VIP has addressed interoperability and reproducibility concerns, in the larger scope of a FAIR (Findable, Accessible, Interoperable, Reusable) approach to scientific data analysis. By implementing the CARMIN API and by using the Boutiques cross-platform framework for applications, VIP provides interoperability with existing platforms, which contributes to reproducibility. VIP provides us with a strong experience and a solid set of users and applications based on which we will tackle the lack of reproducibility L1, L2 and L3 described above.

In order to reconstruct and interpret medical images, researchers make use of numerous image processing algorithms. Each processing step, from the raw image to the final decision, has its specific parameters and may come from a large number of different software packages and dependencies. As a result, the barrier to entry for non-expert users is high and can easily lead to processing pipelines quickly put together that are non-reproducible. Our final aim is to provide an integrated, end to end solution, allowing researchers to launch reproducible executions in a transparent manner. The proposed solutions for evaluating and improving reproducibility will be integrated in VIP and demonstrated on two scientific use-cases sharing a common set of processing tools for MRI image processing and addressing two different challenges: (i) optimising the MRI acquisition protocol w.r.t. to the signal to noise ratio (SNR) and (ii)optimising a processing pipeline for stroke prediction.