Vanishing Gradient Problem

How Activation Functions Have Evolved

We often attribute the success of deep learning algorithms to the increase in computing power. The fact that we can calculate the gradients of deep neural networks so fast made it a lot more practical to train our models using the backpropagation algorithm.