BIDS Machine Learning and Science Forum — A Differential Geometry Perspective on Orthogonal Recurrent Models

ML&Sci Forum

February 8, 2021
11:00am to 12:00pm
Virtual Participation

BIDS Machine Learning and Science Forum
Date: Monday, February 8, 2021
Time: 11:00 AM - 12:00 PM Pacific Time
Location: Participate remotely using this Zoom link 

A Differential Geometry Perspective on Orthogonal Recurrent Models 

Speaker: Omri Azencot, Ben-Gurion Univeristy
Abstract: Recently, orthogonal recurrent neural networks (RNNs) have emerged as state-of-the-art models for learning long-term dependencies. This class of models mitigates the exploding and vanishing gradients problem by design. In this work, we offer a novel perspective to orthogonal networks by employing tools and insights from differential geometry. We show that orthogonal RNNs may be viewed as optimizing in the space of divergence-free vector fields. Specifically, based on a well-known result in differential geometry that relates vector fields and linear operators, we prove that every divergence-free vector field is related to a skew-symmetric matrix. Motivated by this observation, we study a new recurrent model, which spans the entire space of vector fields. Our method parameterizes vector fields via the directional derivatives of scalar functions, which requires the construction of latent inner product, gradient, and divergence operators. In comparison to state-of-the-art orthogonal RNNs, We show that our approach achieves comparable or better results on a variety of benchmark tasks.

The BIDS Machine Learning and Science Forum meets biweekly to discuss current applications across a wide variety of research domains in the physical sciences and beyond. These active sessions bring together domain scientists, statisticians, and computer scientists who are either developing state-of-the-art methods or are interested in applying these methods in their research. This Forum is organized by BIDS Faculty Affiliate Uroš Seljak (professor of Physics at UC Berkeley), BIDS Research Affiliate Ben Nachman (Physicist at Lawrence Berkeley National Laboratory), Vanessa Böhm and Ben Erichson. All interested members of the UC Berkeley and Berkeley Lab communities are welcome and encouraged to attend. To receive email notifications about upcoming meetings, or to request more information, please contact


Omri Azencot

Assistant Professor, Computer Science, Ben-Gurion Univeristy