TASK 3 MLOPS

PROBLEM STATEMENT:

Machine Learning -DevOps integration

1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile.

2. When we launch this image, it should automatically starts train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins.

4. Job1 : Pull the Github repo automatically when some developers push repo to Github.

5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training (eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the CNN processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : If metrics accuracy is less than 80% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created.

9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left.

Solution :

We will start with the creation of Dockerfile.

Here, I've created a folder task3. Inside this folder we have our Dockerfile.

Now, we will build this Dockerfile using this command:

docker build -t <Name of the image>:<Version name> <Folder where Dockerfile located>

Name of the image : my

Version name : v1

Folder where Dockerfile located : task3

Our image is now ready.

We then move towards the jobs.

Job1: Pull the Github repo automatically when some developers push repo to Github.

For this job, we've used the hooks and post-commit so that developer does not need to upload the same code again and again.

sudo cp -rfv / task3/

Using this command we will copy the entire code of developer inside our operating system.

Job2: By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training (eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the CNN processing).

Job3 : Train your model and predict accuracy or metrics.

This job will run only if job2 build is stable.

Job4 : If metrics accuracy is less than 80% , then tweak the machine learning model architecture.

For this we will create this loop:

Job5 : This is our notifier job for best model is created. For this I have used python scripts to send email.

import smtplib

from email.mime.multipart import MIMEMultipart

from email.mime.text import MIMEText

sender_address = "devmaud41122@gmail.com"

sender_pass = "*****"

receiver_address = "mansid@gmail.com"

subject = "Regarding Success of your model "

content = '''Hello, 

This is your best accuracy. Your model is trained successfully with a good accuracy.

THANK YOU ...'''

message = MIMEMultipart()

message['From'] = sender_address

message['To'] = receiver_address

message['Subject'] = subject

message.attach(MIMEText(content, 'plain'))

session = smtplib.SMTP('smtp.gmail.com', 587)

session.starttls()

session.login(sender_address, sendde_pass)

text = message.as_string()

session.sendmail(sender_address, receiver_address , text)

session.quit()

print('Mail sent successfully')

Now, due to some reason if our container fails then whole setup will go. So, for this we have our job6.

Job6 : In build, execute shell we will write:

if docker ps -a | grep python_code

then

if docker ps | grep python_code

then

echo "Everything is working fine"

else

docker start python_code

fi

else

docker run --name python_code hw2:v1

fi

So, this was all about my task3 of MLOPS.