Integration of Machine Learning and Devops

It is found that 60-90% of Machine Learning and AI models fail in real-world ,reason is a lot of manual work for the Data-scientist to manipulate Architecture of the model to get Better and better accuracy.

Here comes the concept of MLOps,in which we integrate the machine learning with Jenkins and Docker as Devops tools to automate the whole manual part from model training to testing and Deployment.

Explanation of task step by step

1.Create container image that’s has Python3 and Keras or numpy installed using dockerfile

2. When we launch this image, it should automatically starts train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins

4. Job1 : Pull the Github repo automatically when some developers push repo to Github.

5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created

9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left

Implementation Work:

Creating the code of machine learning:

• Here I build the CNN architecture.we are gonna run this code in the docker container so this is required to attach/link a volume so that we can fetch the accuracy.txt file .

Creating a docker image:

• For this we have to create a Dockerfile and build it with #dockerbuild command.

#docker build -t automl:v11 .

For Reference go to my git-hub :

https://github.com/17ejtcs031/ml_task3.git

Create jobs in Jenkins:

Job1 - download code and copy it to the workstation(attached volume)

Job2 - check code nature(here it only work for CNN codes) and launch docker image which automatically start training the model.

Job 3 - if we don't get desired accuracy then tweak the architecture of the model and train it again.

job 4 - if we got desired accuracy then this job send an e-mail to notify this.

job 5 - it monitor the job 2 , if any case our docker container goes down or fail then it restore it again and follow the whole process.

Visualization of all the jobs using Build pipeline:

Now everything is configured as soon as developer push the code it automatically generate the model with desired accuracy and save it for the future use.