On May 21–22, BIDS hosted a Reproducibility Workshop. We had about 35 participants, including several BIDS fellows and staff as well as representatives from our NYU and UW partners. In addition, there were students, faculty, and staff from numerous departments and organizations from all over the campus.
Reproducibility Case Studies
On Thursday, we began with a few introductory remarks and then got straight to work writing up individual case studies describing concrete workflows that demonstrate the "how" of reproducible research. These case studies are part of a larger effort led by Justin Kitzes, a BIDS fellow, to gather several dozen case studies from individual researchers from different fields. These case studies are intended to describe the computational workflows used to complete a single well-defined research output, such as a manuscript or software package. Each case study consists of a schematic diagram and an accompanying narrative. After working individually, several participants presented their case studies. While the case study reports are first drafts, they are already useful, and I recommend spending some time reading them.
Reproducibility Courses
On Friday, we had two main sessions: "Education and Curriculum" and "Reproducibility Self-Assessment." The self-assessment session was mainly a brainstorming session, and the BIDS reproducibility working group will be pursuing this further, so you can expect to hear more about this later. For now, I will focus on the "Education and Curriculum" session.
Just a few years ago, you would have been hard pressed to find more than a handful of courses focused on teaching computational reproducibility practices. However, this type of course is becoming increasingly common as the need for this type of training becomes more apparent. For example, this fall I will be teaching a new course titled "Reproducible and Collaborative Statistical Data Science.”
We began by briefly discussing our experiences attempting to teach computational reproducibility. In addition to my discussion of my past experience and plans for the coming semester, Rachel Slaybaugh discussed her experience teaching "Putting the Science in Computational Science," and Randy LeVeque discussed his experience teaching version control in his “High-Performance Scientific Computing” course.
After that, we broke up into small groups to compare course syllabi, philosophies, and strategies for teaching computational reproducibility. My group focused on assessment. In particular, we discussed the necessity of assessing the process and not just the product or end result of student work while recognizing the challenge of providing high-quality feedback to increasing numbers of students as class sizes grow. You can find notes from our discussion here.
Reproducibility Tools
In addition to the three main sessions, we also had a number of lightning talks spread out over the two days. Many interesting tools and approaches were presented, but for the sake of brevity, I will only mention nbgrader—a system for assigning and grading IPython/Jupyter notebooks. Like many people, I've developed my own system (gradebook) to automate assigning, collecting, grading, and recording homework assignments (I use it for group projects and exams as well). My system is fairly lightweight (~350 lines of Python code) compared to nbgrader (~7,000 lines of Python code, ~7,000 lines of CSS, and ~10,000 lines of Javascript). I also had different design goals (e.g., since I have been teaching students to develop R and Python packages, I want students to submit their work as R or Python files in a specific directory layout rather than as notebooks). However, writing gradebook as well as helping other people use it has made me appreciate the need for something like nbgrader. Jess Hamrick started nbgrader for UC Berkeley's "Computational Models of Cognition” (COGSCI 131) course. She has written up her experience developing and using nbgrader here. Jess is actively developing nbgrader, and I suspect that over the next year, it will rapidly grow in popularity. If you are teaching a class that involves computational ideas but isn't focused on teaching programming, you may want to take a close look at nbgrader.