Machine Learning Automation

90% of Machine Learning models never come into production environment . As enterprises are learning the significance of AI changing the way of doing business, there are hurdles along the way causing churn in AI adoption. 

This project uses jenkins for job creation, tensorflow for deep learning and creating neural network , docker for creating container and dockerfile to build images , python to integrate the workflow and to query using jenkins jobs.

Prerequisites:

1. User has github account with git bash installed
2. Virtual Machine with docker and jenkins configured
3. Deep learning model

Task Description With Followed Steps:

Step 1: Build the container images from dockerfiles created that has all required libraries

This file has set-up for deep learning and all required libraries for deploying it to production environment.

This file has set-up for simple machine learning model to deploy on production environment.

Build the docker images as:

docker build -t ml_CNN
docker build -t ml_simple

We can push this image to dockerhub as well so that other users can use it.

Step 2: For automating pushing to github ,build a post-commit hook

Step 3 : Create jenkin jobs

JOB-1:Pulls the Github repository automatically when a developer pushes repository to Github.

JOB-2: By looking at the code or program file, Jenkins will automatically start the respective machine learning software ,installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins will start the container that has already installed all the softwares required for the cnn processing).

This job will trigger only when job-1 is successful.

This will copy the code in the respective directories

JOB-3:Train the model and predict accuracy or metrics.

This job will trigger after job-2 is successful.

JOB-4: If metrics accuracy is less than 80% , then will tweak the machine learning model architecture.

This job will trigger after job-3 is successful.

Add a post-build action for email notification.

JOB-5: Monitor the container for successful training

The job structure appears as:

The build pipeline after successful completion of all jobs

Deep Learning Code Snapshot:

Code Link : https://github.com/shiviagarwal21/ml-devops