Best practices for Product testing on Cloud infrastructure

"Stop Testing"! This is the exact word of caution we got from our leadership Team. Of course I was also one among them and had to agree with it. This happened when we migrated from inhouse infrastructure to Cloud Infrastructure in 2015. That time we were newly adapting to cloud revolution.

Our Operational cost suddenly increased to thrice, as we were using the cloud infra same as our inhouse infra. Earlier we had the Luxury of running Manual functional test separate, Automation regression separate. CIT & SIT are separate and Non-functional testing separate. That time we all told and marketed about cloud infra on the luxury of creating systems in a minute but it has not been highlighted that every minute of usage is going to cost the money. We used to create separate systems for each type of our testing needs and used as per our convenience. Since it was a global R&D company, there were no Time restrictions. So testers were using them at their convenient times round the clock. When we got the first bill from the cloud infra provider then we all realized and restricted the usage. Test teams lost their freedom of time, freedom of infra usage and freedom of R&D in testing.

We reworked on our strategy and implemented few Best practices in the usage. I've refined few of them further and sharing it here:

To optimize testing on cloud infrastructure effectively, we can focus on cost-efficiency, resource management and automation.

> Utilize cloud cost monitoring tools to track expenses in real-time. Set clear budget limits and establish cost-awareness within the testing team.

>Regularly review and optimize costs by rightsizing resources and eliminating or switching off unused resources. Make sure Auto start ON & OFF script is part of your Automation framework

>Continuously assess testing resource requirements and choose appropriate instance sizes. I would suggest to have only micro & mini provisions. And Use auto-scaling to adapt resources to changing workloads, avoiding over-provisioning.

>Consider serverless options for specific testing scenarios to pay only for actual usage.

>Implement DevOps practices to automate resource provisioning, configuration, and deployment.

>Automate testing As much as possible, so that we can reduce to have multiple & dedicated systems for each individual user.

>Establish and enforce cost policies, alerts, and limits to prevent budget overruns. Ensure that the testing team is aware of and complies with these policies. Regularly review and adjust policies based on evolving testing needs. This will avoid the frustration in test teams and can bring up some discipline in using the test infra. And most importantly the awareness that the Management is not asking us to Stop testing, but to well utilize the infra!

>Containerize testing components using technologies like Docker. Implement container orchestration (e.g., Kubernetes) to efficiently manage and scale containers.

>We can also explore cloud provider discount programs, such as reserved instances, to reduce costs for long-term testing projects. Utilize spot instances for non-critical, fault-tolerant testing workloads at lower costs.

>Test data also takes good amount of storage. We can optimize the storage cost by Archiving or deleting unnecessary test data. And if the product architecture permits we can also use cloud-native storage solutions which offer cost-efficient options.

By combining these strategies, you can ensure that your testing on cloud infrastructure is both cost-effective and highly efficient, allowing your testing team to focus on delivering high-quality results while minimizing operational expenses.

Regards.. -Ajay