Integration of ML with DevOps.

Machine Learning is the trend in the present world and inside machine learning, there is Deep Learning there we use the concept of the perceptron, hyperparameters, neutrons, layers, etc. There are different types of Neural Networks like ANN, CNN, RNN.

The main problem in the neural networks is Hyper Parameters. Hyper Parameters are those parameters that are assigned before the building of the model like the number of filters, size of the filter, number of neurons, number of epochs, etc. We have to assign them manually.

So to deal with this problem we can automate it where the hyperparameters will change and we will get the desired accuracy of the model. Here, I used Github, Jenkins, and Docker for automation purposes.

In this CNN model, I used "mnist" dataset to build a CNN python code, then through automation, the hyperparameters will change and I desire accuracy more than 97%. The complete code will run on containers launched by dockers.

Process Flow:

Step 1: First of all, I create a container image that has Python3 and Keras installed in it, using a Dockerfile. I have created the image named "keras_image:1" from this Dockerfile.

# docker build -t keras_image:1 /task3/

Step 2: When we launch this image, it will automatically start to train the model in the container.

Step 3: Create a job chain of 5 jobs using the Build Pipeline plugin in Jenkins.

Job 1: Pull the Github repository automatically when the developer pushes the repository to Github. By using github-webhooks, it will pull the code from Github to Jenkins Workspace. For this, there is a need for tunneling. I used "ngrok" for this purpose.

My GitHub repository is: "https://github.com/Rishi964/mlops_project.git".

Job 2: Jenkins will launch the container with the help of that customized image, we created. It will run only if Job1 was successfully built.

Job 3: It will train the model and predict accuracy in the file "accuracy.txt". This job will run after successfully completion Job 2.

We had 2 dense layers and 3 epochs in this model and we got an accuracy of 96.28%. But we desire more than 97%. So, our main task starts from here. Let's proceed.

Job 4: If accuracy is less than 97%, then tweak the machine learning model architecture. This job will after the successful built of Job3.

Here, we increase one more dense layer and increase 5 more epochs to 8 epochs.

Now, we got an accuracy of 97.07% which was desired (greater than 97%).

Job 5: This will send an email to the developer about the successful completion of the CNN training model that achieved the desired accuracy. This job will run after the successful built of Job4.

Job 6: Create one extra job to monitor. If the container where the app is running fails due to any reason then this job should automatically start the container again from where the last trained model left.

Now, our task is completed. And the complete pipeline will look like this.

The project ends here. Thanks for reading.