Equivariant and Coordinate Independent Convolutional Networks: A Gauge Field Theory of Neural Networks

What is the appropriate geometric structure for neural networks that process spatial signals on Euclidean spaces or more general manifolds? This question takes us on a journey which leads to a gauge field theory of convolutional networks.Feature vector fields: The spatial signals we are interested in are fields of feature vectors. Feature fields allow to describe data like images, audio, videos, point clouds, or tensor fields, such as fluid flows and electromagnetic fields.Equivariant networks commute with actions of some symmetry group on their feature spaces. The relevant group actions in this work are geometric transformations of feature fields, like translations, rotations, or reflections of images. Equivariant models generalize everything they learn over the considered group of transformations. This property makes them significantly more data efficient, interpretable, and robust in comparison to non-equivariant models.Convolutional Neural Networks (CNNs) are the most common network architecture for processing feature fields. Conventional CNNs operate on Euclidean spaces and are translation equivariant, i.e. position independent. This work explains how to extend CNNs to be equivariant under more general symmetries of space.Coordinate independence: Manifolds are in general not equipped with a canonical choice of coordinates. Feature fields and neural network layers are hence required to be coordinate independent, that is, expressible relative to different frames of reference. The ambiguity of local frames represents the gauge freedom of our neural field theory. We show that the demand for coordinate independence requires CNNs to be equivariant under local gauge transformations.To offer an easy entry, the first part of this work focuses on the representation theory of equivariant convolutional networks on Euclidean spaces. The insights gained in the Euclidean setting are subsequently leveraged to develop the full gauge theory of coordinate independent CNNs on Riemannian manifolds. In the last part, we turn to a discussion of practical applications on specific manifolds. A comprehensive literature review demonstrates the generality of our theory by showing for more than 100 models from the literature how they can be understood as specific instantiations of ’Equivariant and Coordinate Independent CNNs’.