Introduction to Deep Learning

Deep Learning refers to the training of large neural networks (with multiple hidden layers) using vast amounts of training data.

What is a Neural Network?

Neural Networks are Machine Learning models inspired by the vast interconnection of biological neurons in the human brain. These networks consist of input and output layers, and one or more hidden layers. Each layer contains several nodes/neurons. Every node in the input layer is connected to every node in the hidden layer, and so on.

Deep Neural Networks have multiple hidden layers.

Neural Networks are incredibly good at mapping the input variables x to the target variable y, if given enough training data and sufficient information about x and y.

The image below depicts a simple neural network to predict house prices, given certain features of the house:

Supervised Learning with Neural Networks

Supervised Learning is a type of Machine Learning paradigm wherein the output labels are known in advance i.e. before we train the model. There are mainly two Supervised Learning techniques:

• Classification: The target variable has discrete values (yes/no, 0/1, etc.)

• Regression: The target variable has continuous values (home prices, etc.)

Some common applications of Supervised Learning techniques:

Input (x)	Output (y)	Application	Type of Neural Network
Home Features	Price	Real Estate	Standard NN
Ad, User Info	Click on Ad? (0/1)	Online Advertising	Standard NN
Image	Object (1, ..., 1000)	Photo Tagging	Convolutional NN
Audio	Text Transcript	Speech Recognition	Recurrent NN
English	Chinese	Machine Translation	Recurrent NN
Image, Radar Info	Position of other Cars	Autonomous Driving	Hybrid/Custom NN

Note: RNNs are mainly used for sequence data/time series data, i.e. data that exhibits a temporal property.

Structured vs. Unstructured Data

Structured data refers to data that has a well-defined meaning and can usually be stored in organized databases. Examples include prices, ages, etc.

Unstructured data refers to data that doesn't have a well-defined meaning. Examples include pixel data, audio data, etc.

Machines usually find it harder to make sense of unstructured data, when compared to structured data, but advances in neural networks are making computers capable of understanding unstructured data as well.

Why is Deep Learning Taking Off?

In the last few years, we have seen a tremendous increase in the amount of data available for training our ML models. The performance of traditional ML algorithms increases upto a point, when we increase the amount of data, but then plateaus because they are not able to take advantage of the large amounts of training data.

For Neural Networks, however, the performance increases as either of the following increase:

• Size of the training data

• Size of the Neural Network

As depicted by the following graph, we can see a change in performance only when there are large amounts of data, because for small amounts of data, performance depends more on feature engineering than the model architecture.

Another important factor that has led to improved performance of ML models is the improvements made to training algorithms. For example, using the ReLU activation function instead of the Sigmoid activation function speeds up the training of neural networks. Also, advances in GPU computing have accelerated the training of efficient Deep Learning models.

To summarize, the main drivers of Deep Learning success are:

• More data

• More efficient computation

• Better algorithms