Logistic Regression

Logistic Regression is one of the most common machine learning algorithm for binary classification.

Let’s see how it works!

For Training part, first we try to find a line/solution that separates the two classes.

•  For this, Logistic regression uses logloss function to calculate the error. The machine at begin will randomly draw the line that classifies the two classes. Then logloss function will impose very small penalty on correctly classified points and large penalty on misclassified points. The total error will be high if there are lot of misclassifications.
•  Then just like in Linear Regression, we use Gradient descent to decrease this error. The line will be moved in some direction and again logloss will calculate error, if error is less than previous it will move in same direction otherwise it will move in opposite direction i.e increase/decrease weights of slope and y-intercept.

These steps will be repeated again and again until the error is minimized.

Once the error is minimized, we will have a best fit line that separates datapoints of two classes.

Then this line will be converted into a curve using a function known as logistic function or Sigmoid function. Any datapoint having any value will be brought in between (0-1), which represents the probability. The curve will be move in a direction such that every datapoint will have maximum likelihood belonging to their respective classes.

If the probability of a datapoint ≥ 0.5, it will belong to class 1 otherwise it will belong to class 0. 

For Testing part, the new datapoint will be passed through sigmoid function and will obtain probability and based on maximum likelihood it will be classified into a class.

Just like binary classification, logistic regression can be used for Multiclass classification too.

We divide this problem into multiple binary classification problem.

It works on one vs rest concept. It will group the classes into two category and then find the separating line for it. For every iteration, different combination of one class vs all class will be divided into two category and machine will find the separating solution. In every iteration a new independent model will be created. 

And while testing, the input data will be passed through every model and the output will be calculated. The output which has highest value will be considered as our output. The total number of outputs in a multinomial logistic regression problem is equal to total number of classes.

Suppose there are three classes A, B and C. Three models will be created from :-

1) A vs (B, C)

2) B vs (A, C)

3) C vs (A, B). 

And a new datapoint will be passed through each model and we will get three output. The value which will be highest/maximum among the three will be selected.