Dr. Marcel Böhme
Lecturer (a.k.a. Assistent Professor)
ARC DECRA Fellow
Room 131
25 Exhibition Walk
Clayton VIC 3800
Monash University
- I will educate my graduate students about sound empirical analysis and reproducibility.
- We will implement our techniques directly into the baseline and avoid unrelated changes.
- We will make all our source code publicly available upon acceptance (as far as funder allows).
- We will make all our papers available by Green Open Access (as far as publisher allows).
- We will share data, scripts, and figures for the main results under CC-BY.
- We will add a "Reproducibility" declaration at the end of each paper.
Software Engineering Reproducibility Manifesto (SERM)
Are we in a reproducibility crisis? Many scientific disciplines are. Earlier research findings cannot be reproduced or are not reproducible in the first place. However, the software engineering community has the unique opportunity to provide almost all artefacts that are required for successfully reproducing the entire set of a paper's findings. All our artefacts are digital. For instance, there are some excellent guidelines on how to effectively package all artefacts required for reproducing the findings.
This is a grassroots effort to inspire collective action from the local level. In terms of governance, reproducibility and open science have gained massive support at the community level, as well. For instance, the ROSE Festival celebrates replication in software engineering. The ACM provides artifact badges for repeatable, replicable, and reproducible artifacts that are published together with the article. Flagship software engineering conferences, such as ESEC/FSE'19 have Open Science guidelines, dedicated Open Science chairs, and Artefact Evaluation committees.
The reproducibility of research is at the very heart of the scientific method. In specific instances, it is fairly reasonable to avoid sharing data, scripts, and tools, e.g., if the authors wish to develop and commercialize the evaluated prototype or if the authors are bound by non-disclosure agreements. However, the general attitude towards science should be one of openness and sharing. We should be allowed to inspect and build upon each others findings. If a central claim in the paper is supported by empirical evidence, we should be able to provide that empirical evidence (within reason). If we are uncomfortable sharing a research prototype, what does it mean for the validity of empirical evidence? There is no need to protect. Science is not a competition.
Other links:- Rohan Padhye (UC Berkeley) on constructing good reproducibility packages.
- Brendan Dolan-Gavitt (NYU) on bugs and baselines.
- The ROSE Festival celebrates replication in software engineering.
- The ACM provides artifact badges for repeatable, replicable, and reproducible artifacts.
- Richard Feynman on what constitutes "good" science.
- Inspired by Lorena A. Barba's reproducibility manifesto.