Scientific computing meets big data technology: An astronomy use case

Zhao Zhang, Kyle Barbary, Frank Austin Nothaft, Evan Sparks, Oliver Zahn, Michael J. Franklin, David A. Patterson, Saul Perlmutter 

2015 IEEE International Conference on Big Data (Big Data)
October 29, 2015

Scientific analyses commonly compose multiple single-process programs into a dataflow. An end-to-end dataflow of single-process programs is known as a many-task application. Typically, tools from the HPC software stack are used to parallelize these analyses. In this work, we investigate an alternate approach that uses Apache Spark—a modern big data platform—to parallelize many-task applications. We present Kira, a flexible and distributed astronomy image processing toolkit using Apache Spark. We then use the Kira toolkit to implement a Source Extractor application for astronomy images, called Kira SE. With Kira SE as the use case, we study the programming flexibility, dataflow richness, scheduling capacity and performance of Apache Spark running on the EC2 cloud.

Featured Fellows

Zhao Zhang

BIDS Alum - Data Science Fellow

Kyle Barbary

Berkeley Center for Cosmological Physics

Saul Perlmutter

Faculty Director