Automate Hyper-parameter tuning of ML/DL model to increase accuracy.

Being a Machine/Deep learning developer the real challenge is not model building but making the model more accurate or deciding its architecture that will help in increasing accuracy of our model. Choosing the parameters that needs to be considered and some more challenges.

Have you ever thought how easy it would be if we just have to create the model and our machine can handle the rest of the hustle which means that it will test our code itself and check its accuracy, if it does not meet the bar then it will automatically start working on it and increase its accuracy. This is the beauty of MLOps, using it we can actually achieve this level of automation and make it easy for all ML/DL developers out there.

I have created a project which performs all the tasks that I have described above and helps us automate all this process. All we have to do is to just upload/push our code/model that we have developed. It will automatically download the code from github, test it and launch a container with suitable running environment according to the needs or dependencies of the model. Thereafter, it will test the code, record its accuracy and work on increasing its accuracy if required automatically. After all this is done, it will keep on monitoring this docker container and make sure that it does not goes down, if this happens restart the container immediately.

Here is a basic overview of how I have created this project.

1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile 

2. When we launch this image, it will automatically starts train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins 

4. Job1 : Pull the Github repo automatically when some developers push repo to Github.

5. Job2 : By looking at the code or program file, Jenkins will automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created

9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left

GitHub link : https://github.com/BlackHat-TG/MLOps_auto.git