Learning from a few examples is challenging for most deep learning systems. In this talk, I will describe recent efforts on advancing this regime through forms of "meta-learning". I'll begin with a brief overview of seq2seq models (which we introduced in mid 2014), and pointer networks (from mid 2015). From the pointer networks framework I'll derive Matching Networks, an architecture which achieves good accuracy on one-shot learning setups applied both to OmniGlot and Imagenet. I'll also describe two similar efforts which employ memory and learning-to-learn aspects to this challenging learning regime.

Unsupervised learning, in particular learning general nonlinear representations, is one of the deepest problems in machine learning. Estimating latent quantities in a generative model provides a principled framework, and has been successfully used in the linear case, e.g. with independent component analysis (ICA) and sparse coding. However, extending ICA to the nonlinear case has proven to be extremely difficult: A straight-forward extension is unidentifiable, i.e. it is not possible to recover those latent components that actually generated the data. Here, we show that this problem can be solved by using temporal structure. We formulate two generative models in which the data is an arbitrary but invertible nonlinear transformation of time series (components) which are statistically independent of each other. Drawing from the theory of linear ICA, we formulate two distinct classes of temporal structure of the components which enable identification, i.e. recovery of the original independent components. We show that in both cases, the actual learning can be performed by ordinary neural network training where only the input is defined in an unconventional manner, making software implementations trivial. We can rigorously prove that after such training, the units in the last hidden layer will give the original independent components. [With Hiroshi Morioka, published at NIPS2016 and AISTATS2017.]

This talk gives an overview of the work we did in the last two years on low-level image processing, focussing on image super-resolution (SR). The starting point is the efficient sub-pixel convolutional network (ESPCN), a statistically motivated deep architecture that achieved state-of-the-art results as measured by peak signal-to-noise ratio (PSNR) or MSE. However, MSE is known to be a very poor measure of perceptual quality and I argue that in low level image processing thinking about the loss function is far more important than thinking about architectures or models. Our next technique, SRGAN used a combination of adversarial training with deep feature matching and achieved massive improvements in perceptual quality. In even more recent work, AffGAN, we established mathematical connections between adversarial training and amortized variational inference. I will finish the talk with an overview of a general class of adversarial techniques for variational inference, of which image SR is a special case.

Sequence to sequence models have recently been used successfully in many disparate application areas, such as machine translation, text recognition, and automated speech recognition. These architectures generally include an encoder for the input sequence, a decoder for the output sequence, and an attention mechanism that couples the two. In this talk we describe a sequence to sequence model for lip reading. It is able to recognize phrases and sentences being spoken by a talking face, with or without the audio, in an open world scenario with unrestricted vocabulary. An important aspect of training such models is the availability of a large dataset of aligned input and output sequences. We describe how the dataset was built automatically in this case from TV broadcasts, using a form of self-supervision for the alignment. The trained model can indeed read lips, with a performance that exceeds human ability, and we also show that lip reading can improve the performance of automated speech recognition. This work is with Joon Son Chung, Oriol Vinyals and Andrew Senior.