Neural Networks

These models aim to mimic the human brain, or are at least inspired by it.

The basic unit in a neural network is a neuron. A neuron computes a weighted sum of its inputs and then an activation is computed.

$X=\begin{bmatrix}x_0\\x_1\\.\\.\\.\\.\\x_d\end{bmatrix}$, $W=\begin{bmatrix}w_0\\w_1\\.\\.\\.\\.\\w_d\end{bmatrix}$and $x_0 = 1$.

The sigmoid activation function $\frac{1}{1+e^{-W^TX}}$ gives a probability as an output. The threshold/step activation function outputs 1 if $W^TX > 0$ and 0 otherwise. The linear activation function simply outputs $W^TX$.

Some common activation functions:

A neural network (also known as a Multi Layer Perceptron (MLP)) has multiple layers of neurons. The most common problem faced in neural networks is the credit assignment problem: it is difficult to determine which neurons are to be given credit/blame for an increase/decrease in accuracy.