Revolutionize Automation Testing With Machine Learning

Introduction:

Now-a-days the buzz word is AI and ML especially LLM. Today we are talking about mostly ML.

What is ML?

Machine learning is a subset of artificial intelligence that enables computers to learn patterns and make predictions or decisions without explicit programming. It involves algorithms that analyse data, identify patterns, and improve performance over time. Common applications include image recognition, natural language processing, and recommendation systems.

So everyone try to use the power of new tech and improvise their work.

Machine Learning in Software testing Space:

Machine Learning we mostly see implementation in product development space. We can hardly find article or very negligible case studies in test space.

While machine learning in software testing is gaining traction, its widespread adoption is not yet universal. Challenges include the need for extensive labelled data, potential model interpretability issues, and the complexity of integrating ML into existing testing workflows. However, its promise lies in automated test case generation, anomaly detection, and adaptive testing strategies, offering potential efficiency gains and improved software quality over time. As the technology matures, its adoption in software testing may increase.

So today we are going to discuss another use case where we can use power of Machine Learning to enhance test capability.

Use case :

Now-a-days all companies use test automation as primary source of quality check. As per my experience every day automation test runs before work hours and test engineers check reports, analyse it and figure out the failures.

Failures may be because of 3 main reasons.

1. Application/ supported APIs are down.
2. Test code issue(Locators are changes or may need to modify the test code).
3. Application issue i.e. a BUG

But considering an medium size organisation average no of test cases will be more then 10K. And if we consider 10% to 15% failures no of failures are around 1500.

Analysing 1 scenarios an average engineer takes around 3 mins. So considering this no the total effort is huge.

Hence this intermediate service is going to reduce this effort and as test engineer always focus for test code failure.

Solution:

Tech Stack used : 
-  ML.NET
-  C#
-  Azure App Service        

We can introduce a module which can do the activity of bucketing the errors.

ML.NET provides many predefined model, which is easy to integrate. Out of those models we are going to use two of them

1. Issue Classification Model: Issue Classification Model in ML.NET categorizes issues based on nature/type using supervised machine learning techniques, trained on labelled dataset.
2. Image Classification Model: Image classification model in ML.NET categorizes images by training on labelled dataset, using algorithms like ResNet, DNN, SVM, etc.

We use supervised learning to make the models understand what is the cause of error and train them with the data. And the training is continuous learning.

For Issue classification Model, input data can be in the format of CSV, excel, Database table etc. But for Image classification we use error images as input to make model understand type of error.

So the final process will be like

Step 1: Any code change made in product triggers test process

Step 2: Automation test code executes on the latest product build.

Step 3: At the end of all test execution we get a test report which holds data about test passed or failed, if failed error message and stack trace with screen shot.

Step 4: This module takes the report and screenshots attached as input and feed into the Classification ML model to predict.

Step 5: The output is an additional report which holds list of errors like Application deployment issue, API issue, Needs Test code fix or a possible Bug. And based on the error type work can be assigned as below.

• Automation engineer only focus on error which need test code fix.
• Manual testers can only focus on possible bug.
• Infra team can be notified about infra issue.

Step 6: If any anomaly found in results or models prediction is not correct or the confidence of model is low, then models can be retrained to make it more accurate.

Below diagram represents the full process flow.

Conclusion:

In conclusion, harnessing the power of machine learning has the potential to revolutionize automation testing practices. By leveraging ML algorithms, organizations can enhance test case generation, improve defect detection rates, and adapt testing strategies dynamically to evolving software environments. While challenges exist, such as data requirements and integration complexities, the benefits of ML-driven automation testing are undeniable. As technology advances and methodologies mature, embracing machine learning will be crucial for staying competitive in the rapidly evolving landscape of software development and testing.

Here we have just discuss a very small use case. Definitely going forward we are gonging to discuss more on these topics.

Thank you All 😊.