Machine Learning Model Automation

This project is based on continuous checking of a machine learning model and after reaching a desired accuracy, a mail will be send to the developer.

A step-by step process for the project is described below:

• First let us create a Dockerfile for creating a container in which we are creating an image with centos (from dockerhub) and other resources as per our requirements.

• Now we have to build this Dockerfile. After that, you will find an image and here we named this image as cnn:v1. Also we have created an image in order to run a python code(send by developer) of traditional model learning method.
• Write a code of any machine learning model which you want to train. Also provide the complete dataset of the model and save the accuracy of the model in a file , here I have created a file named accuracy.txt and with the help of this accuracy you can decide whether to tweak the model or not.

Code and dataset in the below link:

• Create the Jenkins Job in the following manner:

1. JOB1: Creating a directory named mlops-ws and copying all the contents fetched from GitHub in the directory. Successful job will trigger the next job.

2. JOB2: In this job, we are checking if there is a python file in the directory mlops-ws. If yes, then check whether it has any word named keras and count the number of times it is there in the file and save in a variable. Compare this value, if it is greater than zero, then it returns a value one(in shell) and now we used if-else statements for various cases. If value is equal to one then copy all contents into a directory CNN else copy it into sklearn directory and remove the directory mlops-ws(This is done to ensure that job 1 always creates and copy new content since it would be difficult to find python file everytime) and accordingly launch the docker image. It will automatically train the model and accuracy.txt file will be created. Successful job will trigger the next job.

3. JOB3: Now checking the accuracy from the file. If it is greater than 90% then the model is considered as the best model otherwise we have to retrain the model. Successful job will trigger the next job.

4. JOB4: In this job, again we are checking the accuracy from the file. If less than 90 then we are tweaking the file by changing the number of CRP(Convolution-relu-Pooling) and increasing the number of epochs. Successful job will trigger the next job.

5. JOB5: Training the model again and checking the accuracy. If this time the accuracy is more than 90% then send the mail to the developer, otherwise retrain the model.

6. JOB6: This job is only for monitoring whether the container is running or not in case of CNN. We can do the same for sklearn and this is an independent job.

After all the jobs are successful, we can see a pipeline like this:

I tried to automate this as much as possible but still a more dynamic project can be made by modifying this.

Any suggestion would be appreciated.

Thank You !