Overview

Convolutional and recurrent networks

A fully connected network treats every input the same way. It does not know that two pixels next to each other belong together, or that a sentence has an order.

If we know something about the structure of the input, we can build it into the architecture. That is what this week is about.

Pages

Convolution	Filters, volumes, stride, padding
A convolutional network	On MNIST
Recurrent networks	Sequences of variable length
Exercises

Goals

The goal of this week is to understand how neural networks can be used in creative ways to solve image and text classification problems.

Inductive bias

With a large enough network we can approximate any function, so in principle any architecture would do. In practice the architecture decides how much data and how much tuning we need.

Each method makes assumptions about what a reasonable function looks like. That set of assumptions is its inductive bias.

A convolutional network assumes that a useful feature in one part of an image is also useful in another part, and that nearby pixels matter together. A recurrent network assumes that the same rule applies at every position of a sequence.

For a given data set the inductive bias of one method can be much better than that of another. There is no method that works best on every data set.