Task 2 of mlops/Devops

Problem Statement :

1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile 

2. When we launch this image, it should automatically starts train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins 

4. Job1 : Pull the Github repo automatically when some developers push repo to Github.

5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the software required for the cnn processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created

9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left

SOLUTION :

Before rushing to the solution we must know some keywords and some problems in machine learning . They are :-

HYPER-PARAMETER : They are the parameters which we cant determine with the help of any formula or algorithm but they are adjustable we can change them to achieve higher accuracy or different result . Such as no. of epochs , Size of filter , no. of neurons etc.

1)Machine Learning model are good in prediction but the accuracy of the model is a problem , we have to achieve highest accuracy possible but there are many hyper-parameters when they are changed they may or may not affect the accuracy the model they may increase or decrease the accuracy of model .

2)So to achieve a best set of hyper-parameters is so hard for humans and it is very time and resource consuming , So we have created an architecture that will do this work fully automated .

Now going to solution : -

1. So according to problem statement we will create docker images in which we will install the python interpreter and the modules respective for the code . If we have CNN code we will install pyhton , keras , tensorflow and more needed library . If the code is Linear Regression then we will install numpy , sklearn etc.

2. Now we get to our Jenkins now we will start building the job chain :-

Job1: It is the simplest and first job jenkins just have to go to the github and download the code provided by the developer and it will copy the whole code in a folder in workspace .We have our python code also in this folder which we will discuss ahead .

Job4: This is the most critical job of our architecture this will first check the accuracy of the code if the accuracy is less then needed then we change some code or do some modification or we can say tweak the model then we retrain the model then again we check accuracy this loop will continue until we get desired accuracy (i have used only 80% but i recommend to use more ) . We have code changer for each type of model but i will we explain only the CNN code changer .

Job5: This is very simple job it will only notify the developer that model is ready .

Now we can talk about the pyhton code our code is not so great it have very basic python file handling .

Reader.py :

we just used normal file handling first we loaded model.py file then we open it in read mode and store data on ram then searched for some special strings such as in Linear model the word ''Linear" will be there for sure , in KNN code the word ''Kneighbour" will be there for sure similarly in the CNN code the string "keras" or "tensorflow" and "Conv2D" will be there in this way we can identify the code.

code_changer.py :

In this code we used different approach in different type of models we are gonna discuss mainly CNN in this we just played with the hyper parameter we have increased the no of epochs mostly and increased the no. of neurons and changed some other parameters.

thank you so much for reading