The path to reproducibility
The importance of reproducibility has grown considerably due to the significant rise of this retracted literature. One of the primary ways we use to ensure reproducibility is by repeating certain experiments to show that the presented results can be verified or that they are consistent. Moreover, reproducibility has other advantages, such as facilitating error detection, improving transparency in research, or improving the research methodology.
However, as with most things, nothing has only pros; there are cons as well, and they come in the shape of challenges. To ensure reproducibility, we first need code and data availability. The research community is becoming increasingly aware of the criticality of opening the code, and platforms like GitLab or even Docker Hub (where software can be packaged and uploaded) are helping in this direction. Second, the experimentation methodology must be clearly presented to ensure it can be easily replicated. Automation helps in this way, as if the code can be run with a script or by running a docker, we minimise human errors. There is, though, one more problem, the elephant in the room, hardware. In some experiments, the hardware used for the experiments is critical and complicates reproducibility. Having access to High-Performance Computing (HPC) infrastructures, to specialised hardware like GPUs or FPGAs is often limited for most of the community, torpedoing the ability of verifying the work from our peers. Here is where the AI-on-Demand (AIoD) platform can play a relevant role for the community.
What’s AIoD’s goal in this?
One of the goals of the AIoD platform is to enable users to share and access hardware for experimentation. This capability opens the door to several possibilities from the reproducibility point of view. For instance, researchers could run their experiments on HW shared through the AIoD platform and obtain a certificate or similar documentation describing what hardware was used. Hence, it would be feasible to retrieve this very same hardware for future experiments. Alternatively, conferences could coordinate with institutions, e.g., public HPC centres, to reserve hardware for a specific period prior to the event. This would enable those participants willing to do it, to run the experiments on those premises and, again, be able to credit that the experiments were run on such infrastructure.
We are currently working on enabling users and institutions to share their hardware, using plugin-based tools that are compatible with several orchestrators like OpenStack or Kubernetes. We are also devising methods that can be adopted by HPC centres, whose environments are quite complex due to security constraints. Once the hardware is shared, we allow users to request access to part of these resources and deploy their own tools by integrating cool tools like the Infrastructure Manager. Facilitating the publication and sharing of hardware will boost our capabilities to ensure reproducibility, from the hardware point of view to the community.
Addressing Challenges
Of course, we must remember that these initiatives usually have two silent enemies, hardware availability and costs. However, first we must work on maturing these tools, and, as has been proved many times with the research community—so many that it could easily be our motto—if the need arises, we’ll find a way.
Author: Jordi Arjona (ITI)
