Where is the Functional Test Environment? (Part 1)

A three part post covering use of a Functional Test environment in software testing.

Part 1 - Introduction

What has prompted these posts and why I've written them.

Part 2 - What is a Functional Test environment

What a functional test environment is and how it is used within the SDLC and other test environments.

Part 3 - Why do we need a Functional Test environment

Why a functional test environment is needed in relation to other test environments.

Why these posts?

In a number of recent positions and roles, as well as discussions with other Testers and SDETs, there seems to be an increase in the number of projects taking builds directly to Integration Test from the Unit/Dev environment. The overall result of this is higher testing costs and lower functional test coverage which results in unstable integration environments and higher incidents of test-escapes. Stakeholders get, and remain, confused as they cannot understand why costs are so high, and why quality remains stubbornly low despite investing more and more in the project/s.

There are excellent texts and books on test environments - notably by authors such as Dorothy Graham, Rex Black, Erik Van Veenendaal, Isabel Evans, etc. - but they do tend to assume varying degree's of test-related knowledge and bury the information in boring (to non-testers) passages and chapters. Hence, I thought I'd give this single subject a bash. For many testers (especially SDETs) there is nothing new here; but I hope it may help stakeholders, managers and non-test folk understand this topic.

The common fixes

The usual remedies project and senior managers try is;

Developers are told to be more careful and test their code better.

This is probably the most common way managers try to address the issue. I have even heard managers saying 'If developers were more careful we wouldn't need testers anyway!'. This doesn't seem to work and usually results in good developers leaving an organisation as they get frustrated at being blamed for issues that are either not their fault and/or what they perceive as having to do the jobs testers should be doing better.

Testers are told that they must use Test Automation

This is another classic remedy and is largely driven by the marketing of many commercial test automation tools that promise to solve all the test issues while reducing costs and creating better quality products.

In reality, it causes cost overruns and lower quality at best. And at worse, directly causes projects to fail. Management cannot understand why the $100K investment in test automation didn't work and the tools become shelf-ware.

Various cool sounding test systems are tried

TDD (Test Driven Development), BDD (Behaviour Driven Development), Risk-based testing, variously oriented test-pyramid systems. All are tried but seem to fail. No-one can explain why and they seem to work in other organisations...

Etc...

I've seen, and heard of, numerous things tried by project managers and owners but ultimately they fail with varying degrees of repercussions and managers scratching heads not understanding quite why they just can't get test working smoothly.

Unfortunately, the reason test ROI is so poor is usually due to fundamentals and as such making small steps to fix the issue cannot be achieved as it needs a big step back first.

Why a big step back?

While lack of a functional test environment seems increasingly common (along with the consequences), it is difficult to implement in a project/organisation as it is quite a major change and generally into the unknown with no tenable or visible benefits, and a whole host of issues and potential costs involved.

In addition, lack of a functional test environment may not be the only issue. If an organisation is lacking in something as fundamental as a functional test environment they usually have other test & quality related issues as well. Poor/no tool integration, stakeholder and senior management uncertainty as to the role of test, poor/no tests processes and use of technical tools without training by inexperienced non-technical staff are all common issues within Software Testing. Implementation of a functional test environment, therefore, may not only be seen as a costly change but often highlights the need for a host of other changes that can be seen as being caused by having a functional test environment!

So incorporating a functional test environment within an environment is usually a step back and can take some time - and pain - before the benefits are seen and recognised.

What is a Functional Test Environment

This is a common question that is put to me and will wil covered in the next part of these Posting - however, I will touch on it here (mainly because I've not written part 2 yet!). While the answer is simple, explaining it is surprisingly difficult and there is a lack of clear and concise information online.

The short answer is that a functional test environment is where a single project/application is formally tested, in isolation of any other projects/apps that may be part of a bigger system/solution. It is a phase prior to integration testing with other systems/projects it may interface/communicate with.

The longer answer I'll cover in the next part.