Multilayer perceptrons

Examples

Three ways to write a network

A neural network with one input, a hidden layer of four neurons and one output computes \[ \hat y = g^{(2)}\Big(w^{(2)}_{10} + \sum_{k=1}^{4} w^{(2)}_{1k}\, g^{(1)}\big(w^{(1)}_{k0} + w^{(1)}_{k1}x\big)\Big), \] where \(g^{(1)}\) and \(g^{(2)}\) are activation functions, for example \(\mathrm{relu}(z) = \max(0, z)\). Below are three equivalent ways to write it in code.

1. In the expanded form all parameters and activation functions are written out in one expression. With more neurons or hidden layers this quickly becomes annoying to write.
2. In the neuron form the network is visibly made of simple neurons, each with its own input weights and bias.
3. The matrix form is the most succinct and the one used in practice: the biases of a layer form a vector, and the input weights of all neurons of a layer form a matrix.

import numpy as np

def relu(z):
    return np.maximum(0, z)

def identity(z):
    return z

# 1. expanded
def neural_net_expanded(x, w10_1, w11_1, w20_1, w21_1, w30_1, w31_1, w40_1, w41_1, g1,
                        w10_2, w11_2, w12_2, w13_2, w14_2, g2):
    return g2(w10_2
              + w11_2 * g1(w10_1 + w11_1 * x)
              + w12_2 * g1(w20_1 + w21_1 * x)
              + w13_2 * g1(w30_1 + w31_1 * x)
              + w14_2 * g1(w40_1 + w41_1 * x))

# 2. neurons
def neuron(x, b, w, g):
    return g(b + np.dot(w, x))

def neural_net_neurons(x, w10_1, w11_1, w20_1, w21_1, w30_1, w31_1, w40_1, w41_1, g1,
                       w10_2, w11_2, w12_2, w13_2, w14_2, g2):
    hidden = np.array([neuron(x, w10_1, np.array([w11_1]), g1),
                       neuron(x, w20_1, np.array([w21_1]), g1),
                       neuron(x, w30_1, np.array([w31_1]), g1),
                       neuron(x, w40_1, np.array([w41_1]), g1)])
    return neuron(hidden, w10_2, np.array([w11_2, w12_2, w13_2, w14_2]), g2)

# 3. matrices
def neural_net_matrix(x, b1, W1, g1, b2, W2, g2):
    return g2(b2 + W2 @ g1(b1 + W1 @ x))

1

W1 is the \(4 \times 1\) matrix of the input weights of the hidden layer, b1 the vector of its four biases; W2 is \(1 \times 4\) and b2 has one entry.

All three compute the same number.

w = dict(w10_1=0.1, w11_1=1.0, w20_1=-0.5, w21_1=2.0, w30_1=0.3, w31_1=-1.0,
         w40_1=0.0, w41_1=0.5, w10_2=0.2, w11_2=1.0, w12_2=-1.0, w13_2=0.5, w14_2=2.0)
x = np.array([0.7])

b1 = np.array([w["w10_1"], w["w20_1"], w["w30_1"], w["w40_1"]])
W1 = np.array([[w["w11_1"]], [w["w21_1"]], [w["w31_1"]], [w["w41_1"]]])
b2 = np.array([w["w10_2"]])
W2 = np.array([[w["w11_2"], w["w12_2"], w["w13_2"], w["w14_2"]]])

print(neural_net_expanded(x[0], g1=relu, g2=identity, **w),
      neural_net_neurons(x, g1=relu, g2=identity, **w),
      neural_net_matrix(x, b1, W1, relu, b2, W2, identity))

0.8 0.8 [0.8]

One input, one output

The network above with every weight on a slider. Top, the activation of each of the four hidden neurons, \(g^{(1)}(w^{(1)}_{k0} + w^{(1)}_{k1}x)\). Bottom, the output of the network. Every vertical axis follows the range of its curve.

With relu hidden neurons, every hidden neuron is zero on one side of a point and linear on the other, and the output is a sum of four such pieces: piecewise linear, with a kink where each hidden neuron switches on.

Two inputs, one output

With two inputs, each hidden neuron computes \(g^{(1)}(w^{(1)}_{k0} + w^{(1)}_{k1}x_1 + w^{(1)}_{k2}x_2)\).

Four hidden neurons and one output over the two inputs. Each map is scaled to its own range, dark where it is high. With the default heaviside activation, each hidden neuron is 1 on one side of a line and 0 on the other.

If you want to tinker more with small networks, the TensorFlow playground trains them in the browser.