USING A PRE-RELEASE ENVIRONMENT FOR A CLOUD PREVIEW

Do you need a pre-release environment?

When managing software in the cloud, a pre-release environment to give a preview to your customers, allowing them to test drive new features and functionality, as well as validate existing functionality is essential.  Enterprise customers, and all customers for that matter, need to make sure they will not be impacted by an upcoming major release. Today companies like Salesforce.com, Splunk, Tableau, Microsoft (Azure) and Amazon (AWS) provide environments specifically for evaluation, preview and testing.

What is the best process from technical support to manage a pre-release environment?

A pre-release environment may be persistent but the preview of new software should be time bound. When a new version of a release is updated in a pre-release environment, customers must be informed of the date of the release and how long they will have to test and preview new features. You should also provide a release note providing a high-level overview of all the new features implemented. This can be a beta release note but should minimally contain a comprehensive list of features and functionalities impacted.  Most companies don’t want to say “Test” as this implies the customer is somehow doing QA. In reality there are many combinations and corner cases that can be implemented in enterprise software and they cannot all be accommodated during the QA process, so it is prudent for customers to validate their environments are not affected negatively by a release within a non-production environment.  Sending out a status page update, and/or an email, to all participating customers is the best way to communicating the preview is coming soon, as well as live. It's prudent to give customers at least 3 weeks or more notice of the impeding preview, so they can plan for their testing.

What process should be followed to address issues?

If any issues are found they should be reported by the customer using the normal case management system (CRM).  A good practice is to have a flag in your CRM to identify all pre-release cases. You can use a case type or area field to identify the case, and either allow the customer to input during case creation or ask engineers to flag the cases for tracking. This will make reporting the cases simple and easy to make sure no cases go unnoticed. Depending on the amount of time until the GA release, pre-release cases should be treated as the highest priority, in order to make sure R&D has the time to fix, test and build a new release. Support teams should be informed of the priority and how to handle the cases, and this should be a topic in all regular support team meetings. Additionally, this should be discussed with support engineers before every preview to remind the team of the criticality of such issues.

Updating Pre-release with fixes recommendations:

When rolling patches, updates, or builds into the pre-release environment it is critical that you inform customer of the update and what is being updated. Consider you have an organization that has a testing team that is in the process of testing the release, and there is an update that breaks their environment. They will be completely unaware of the changes rolled out. They may spend a couple of days retesting and questioning their process before they report it. Valuable customer time is lost as well as R&D time to fix the issue. When the update is properly communicated, including the date and time and a small synopsis of the bug fixes rolled out, customers will quickly ascertain whether the update caused a regression. The best way to communicate a fix is to use a standard status page update using an application like Atlassian’s  http://status.io and having a pod specifically defined for the pre-release environment. This will help isolate the communication to this environment, and limit the distribution of messaging to subscribers of this pod/environment.

How can you budget for a separate environment for pre-release?

The best way to accommodate the cost of the additional org/pod is to require a subscription. It would be in the vendor’s best interest to include the additional org in one of the subscription or support stacks. It may be best to minimize the cost to net cost rather than trying to gain additional revenue. Once there is a cost associated to the environment the participation is going to drop, and the feedback loop is going to be significantly less. Even when the usage is free there are going to be limits to the participation as additional resources are required to be allocated for testing. A customer may require a project/test plan, as well as assets including testers, equipment and time are required.

Reporting results to R&D:

It is essential to provide a feedback loop on the pre-release participation, usage, defects and status of outstanding issue. Setting up reports using a reporting application like Tableau to show all data points will help collate the data and make informed data driven decisions. Useful data points to report depend on the specific software. Some ideas could be customer name, number of logins, downloads, jobs run, type of assets accessed, etc. R&D teams generally have staff meetings weekly, and having a standing time to present during the preview will ensure to keep different product teams accountable and informed and allow you to get necessary updates for customers including ETAs for resolution.

Customer Confidence:

If a customer faces too many issues in a pre-release environment they are going to lose confidence in the release. It is support’s role to build the customer’s confidence, collaborate with R&D to obtain expeditious resolutions and assure customers that the whole organization is working toward resolving all outstanding issues. Customers will try to stop the release by saying they are not confident, but as long as all the fixes are addressed, we must convince them that the process is working and assure them that similar to these fixes, we will address any additional issues as expeditiously, and with the highest priority, as we have already done.

Go / No-Go:

Last but not least, prior to the GA/Production release, all pre-release issues should be resolved and signed off by the customers that reported the issues. R&D, along with support management and operations, should have a Go/No-Go meeting to make sure the release is sound and all customer issues are resolved. We should never knowingly go into a release where an issue would break a customer. Yes, issues may be uncovered after a release, but we never knowingly rollout a defect. Calling off a release is not optimal, but customers would be more appreciative to know we made sure to protect them from any defects. In my experience, I have had to call off (postpone) releases and have never had an angry customer.

Summary:

Utilizing a pre-release environment to support customers to make sure they are successful after releases is the best way to mitigate impact from major upgrades. Clear and concise communication, both externally and internally, keep customers informed and all internal teams responsible, accountable and informed. Taking this extra step in the cloud rollout process is tried and true by some of the largest cloud vendors. Tracking and reporting issues, and handling them as the highest priority is key to the outcome. Reducing customer impact greatly influences customer confidence, reduces churn, promotes adoption and consumption and provides the platform for expansion. We have to look at each step of the cloud experience holistically to ensure we are providing the best customer experience.