The Purpose of Activation Functions

In this blog, we will learn about Activation Functions and several strategies for selecting the optimal one for your particular deep learning problem statement.

What are activation functions?

Activation functions are used in neural networks to transform the input signal from the input layer to the output layer. The output of an activation function can be used to control the output of a neuron. There are several types of activation functions, each with its own advantages and disadvantages.

What types of activation functions are there?

1. Binary Activation:

Binary activation functions are either on or off, meaning they either allow the signal to pass through or they don't. Binary activation functions are the simplest type of activation function, but they are also the most limited.

2. Linear Activation:

Linear activation functions allow the signal to pass through unchanged. This makes them very easy to work with, but they can also be very limited.

3. Sigmoid Activation:

Sigmoid activation functions squish the input signal into a range between 0 and 1. This makes them easy to work with and easy to interpret, but they can also be very limited.

4. Tanh Activation:

Tanh activation functions squish the input signal into a range between -1 and 1. This makes them easy to work with and easy to interpret, but they can also be very limited.

5. ReLu Activation:

ReLU activation functions allow the signal to pass through unchanged if it is greater than 0, and they block the signal if it is less than 0. This makes them easy to work with and easy to interpret

6. Softmax Activation:

Softmax activation functions are used to squash the output of a neuron so that it is between 0 and 1 and it also ensures that the sum of all the outputs is 1. This is often used in classification problems.