Principles for data analysis workflows

Sara Stoudt, Valeri N. Vasquez, Ciera C. Martinez

arXiv.org
July 17, 2020

Abstract: Traditional data science education often omits training on research workflows: the process that moves a scientific investigation from raw data to coherent research question to insightful contribution. In this paper, we elaborate basic principles of a reproducible data analysis workflow by defining three phases: the Exploratory, Refinement, and Polishing Phases. Each workflow phase is roughly centered around the audience to whom research decisions, methodologies, and results are being immediately communicated. Importantly, each phase can also give rise to a number of research products beyond traditional academic publications. Where relevant, we draw analogies between principles for data-intensive research workflows and established practice in software development. The guidance provided here is not intended to be a strict rulebook; rather, the suggestions for practices and tools to advance reproducible, sound data-intensive analysis may furnish support for both students and current professionals.

2020-0717 - Stoudt-Vasquez-Martinez - Schematic of a conceptual data analysis workflow - 700
Schematic of a conceptual data analysis workflow.


Featured Fellows

Sara Stoudt

Statistics
BIDS Alum - Data Science Fellow

Váleri N. Vásquez

Energy and Resources Group
BIDS Alum – DATA SCIENCE FELLOW

Ciera Martinez

Biology and Environmental Sciences Lead