Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, Liang-Chieh Chen

Generative image models encode images as tokens in a compact latent representation. For example, a 256x256x3 pixel image is typically broken in to 8×8 pixel patches, and encoded as 32×32 = 1024 16-dimensional vectors. The standard approach assumes that the 2D arrangement of patches is important for tokenisation.

In this paper, motivated by previous successes of 1D models applied to 2D images, the authors replace the 2D patch encoding with a smaller 1D latent representation, which is trained using a Vision Transformer model ^[3].

The 16-dimensional tokens are encoded as 10-bit integers using a vector quantisation (VQ) algorithm with a learnable code book. The authors show good performance with a 1D latent space of only 32 tokens, giving up to a 400x speedup for image generation over current models.

^[3] Taming Transformers for High-Resolution Image Synthesis (2020), https://arxiv.org/abs/2012.09841

Leonardo Defilippis, Bruno Loureiro, Theodor Misiakiewicz

Recent work has shown the surprising result that overparametrised neural networks (with number of features p > number of data points n) exhibit “double descent” ^[4], and can achieve both zero training set error and low test set error. This phenomenon is also present in simple linear models e.g. random feature ridge regression (RFRR).

In this paper, the authors consider RFRR and derive a formula for the test set error as a sum of bias and variance terms. They identify different regimes where either bias or variance dominate. Finally, they improve the lower bound on the minimum number of features required from p* = O(n) to p* = n^q* where the exponent q* depends on the power law distribution of the features and data. Their formula shows impressive agreement with both simulated data and real-world data from the FashionMNIST dataset.

^[4] Two models of double descent for weak features (2019), https://arxiv.org/abs/1903.07571