Deep Learning

Neural Networks went out of fashion in the 90's when SVMs were introduced.

However, with:

• more data

• more processing power

• better algorithms

neural networks came back, in the form of Deep Learning, i.e. learning with neural networks that have several hidden layers.

The first breakthrough was by a team that used a Deep NN to win an Image Processing competition (ILSVRC).

The main advantage of Deep Learning is that we no longer need to specify the features of the data to the model. The model learns the features automatically. Initial layers leran low-level features while layers further in the network learn higher-level features.

Disadvantages: not intuitive, bad interpretability

Convolutional Neural Networks (CNNs)

These were one of the first Deep NN's to be introduced. They have the following types of layers:

• Convolutional Layer: convolves the image with a sliding filter

• Pooling Layer: performs dimensionality reduction

CNNs, like standard NN's, are trained using backpropagation.

The ReLU activation function is used most often. ReLU(x) = max(x,0).

Droupout Regularization can be used to reduce overfitting.