On May 21–22, BIDS hosted a Reproducibility Workshop. We had about 35 participants, including several BIDS fellows and staff as well as representatives from our NYU and UW partners. In addition, there were students, faculty, and staff from numerous departments and organizations from all over the campus.
Reproducibility Case Studies
On Thursday, we began with a few introductory remarks and then got straight to work writing up individual case studies describing concrete workflows that demonstrate the "how" of reproducible research. These case studies are part of a larger effort led by Justin Kitzes, a BIDS fellow, to gather several dozen case studies from individual researchers from different fields. These case studies are intended to describe the computational workflows used to complete a single well-defined research output, such as a manuscript or software package. Each case study consists of a schematic diagram and an accompanying narrative. After working individually, several participants presented their case studies. While the case study reports are first drafts, they are already useful, and I recommend spending some time reading them.
On Friday, we had two main sessions: "Education and Curriculum" and "Reproducibility Self-Assessment." The self-assessment session was mainly a brainstorming session, and the BIDS reproducibility working group will be pursuing this further, so you can expect to hear more about this later. For now, I will focus on the "Education and Curriculum" session.
Just a few years ago, you would have been hard pressed to find more than a handful of courses focused on teaching computational reproducibility practices. However, this type of course is becoming increasingly common as the need for this type of training becomes more apparent. For example, this fall I will be teaching a new course titled "Reproducible and Collaborative Statistical Data Science.”
We began by briefly discussing our experiences attempting to teach computational reproducibility. In addition to my discussion of my past experience and plans for the coming semester, Rachel Slaybaugh discussed her experience teaching "Putting the Science in Computational Science," and Randy LeVeque discussed his experience teaching version control in his “High-Performance Scientific Computing” course.
After that, we broke up into small groups to compare course syllabi, philosophies, and strategies for teaching computational reproducibility. My group focused on assessment. In particular, we discussed the necessity of assessing the process and not just the product or end result of student work while recognizing the challenge of providing high-quality feedback to increasing numbers of students as class sizes grow. You can find notes from our discussion here.