Hadi Vafaii, Dekel Galor, Jacob L. Yates

The authors of this paper are interested in the correspondence between neural networks and biological brains. They argue that variational autoencoders (VAEs) are a promising model for perception in the brain because they incorporate the idea of perceptual inference and learn representations similar to the cortex.

However, they possess some key differences; in particular, biological neurons fire discretely, with the rate of firing thought to encode information, while VAEs typically use continuous distributions in the latent space.

This motivates them to create the Poisson Variational Autoencoder (P-VAE), which uses the discrete Poisson distribution in its latent space. Their main contribution is handling the problem of performing inference over discrete latent Poisson-distributed variables: they consider the Poisson distribution of rate x by counting the number of observations in unit time of a stochastic process with wait times distributed exponentially with mean 1/x, and by replacing the hard threshold function with a sigmoid.

They derive the resulting loss function and show its similarity to the loss function of sparse coding, with a specific ‘metabolic cost term’ penalizing firing rates.

In practical experiments on the Van Hateren natural image database, CIFAR 16×16, and MNIST, they observe that the P-VAE:

• (i) learns basis vectors similar to sparse encoding
• (ii) avoids posterior collapse
• (iii) learns sparse representations
• (iv) is more sample efficient in downstream tasks than Gaussian-based VAEs
• (v) learns representations with higher dimensional geometry

This work shows that some of the shortcomings in using VAEs to model biological neurons can be overcome.

Tycho F. A. van der Ouderaa, Mark van der Wilk, Pim de Haan

Many physical systems exhibit symmetries. When modelling the behaviour of such systems using neural networks, generalisation and overall performance are often improved by incorporating such symmetries; a classic example being how CNNs incorporate translational symmetries.

This paper proposes a new method to learn these by parameterising symmetries in Hamiltonian machine learning models and connecting them with conserved quantities via Noether’s theorem.

Through an ODE flow, each conserved quantity generates a one-dimensional subgroup of diffeomorphisms to which the Hamiltonian is invariant. Restricting to quadratic conserved symmetries, the authors find a closed form for this flow in terms of a matrix exponential. They use this flow to create the orbits of the group action, and symmetrise observables by approximately integrating over these orbits via a Monte Carlo simulation.

With this in place, they use the marginal likelihood of the data given parametrized symmetries to fit models. The marginal likelihood has to strike a balance between fit to the data and model complexity, favouring simpler symmetric models and giving rise to an Occam’s razor effect.

Since the marginal likelihood is generally intractable in closed-form, they use Bayesian inference to get a lower bound that incorporates the symmetrised Hamiltonian and use this as a loss function.

Using matrix normal posteriors factorised per layer to reduce the number of variational parameters, they train models to different physical systems with known symmetries, finding that their model recovers the correct symmetries and outperforms non-symmetric models in generalization.

The paper shows that when modelling Hamiltonian systems, one does not have to choose between imposing or not imposing symmetries on a model, but that the symmetries can be learned, both in theory and in practice.