Sizing Neural Network Experiments

BIDS Senior Fellow Gerald Friedland will present this week's NERSC Data Seminar at Lawrence Berkeley National Laboratory.

Berkeley Lab – CS/NERSC Data Seminar
Date: Friday, April 12, 2019
Time: 12:00pm – 1:00pm
Location: 3101 Shyh Wang Hall (59-3101), LBNL
Seminar Host: Aydin Buluc, Computational Research Division, Lawrence Berkeley National Laboratory

Abstract: Most contemporary machine learning experiments are performed treating the underlying algorithms as a black box. This approach, however, fails when trying to budget large scale experiments or when machine learning is used as part of scientific discovery and uncertainty needs to be quantifiable. Using the example of Neural Networks, this talk presents a line of research enabling the measurement and prediction of the capabilities of machine learners, allowing a more rigorous experimental design process for machine learning experiments. The main idea is taking the viewpoint that memorization is worst-case generalization. My presentation is made of three parts. Based on MacKay's information theoretic model of supervised machine learning~\cite{mackay2003}, I first derive four easily applicable engineering principles to analytically determine the upper-limit capacity of neural network architectures. This allows the comparison of the efficiency of different architectures independent of a task. Second, I introduce and experimentally validate a heuristic method to estimate the neural network capacity requirement for a given learning task. Third, I outline a generalization process that successively reduces the capacity starting at the memorization estimate. I conclude with a discussion on the consequences of sizing a machine learner wrongly, which includes a potentially increased number of adversarial examples.