Automatic UI Testing

Wouldn’t it be nice if we could have 100% test coverage writing fast, stabile unit-tests? Well, that is what we should strive for, but reality often makes this hard; some aspects of a program just need to be tested in a non-unit test context. That said, I have seen too much code that due to its structure couldn’t be unit tested, and as a consequence people resort to other kinds of testing. In other words, they go up in the Test Automation Pyramid (https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid) and start to write more expensive tests – what they should do is to refactor the production code to make it suitable for unit testing.

Let us assume we have structured the production code perfectly such that code is tested at the right level, and still some aspects can only be tested at the UI level. How do we proceed? The two most important goals to reach are speed and the absence of brittleness. These goals are important for all kinds of testing, they are typically just harder to achieve when doing UI testing.

Regarding speed, just as your customers would appreciate the production code to run fast, you will also appreciate fast code when running tests, thus both you and the customers will be happy and it is a kind of a free lunch! But eventually you will reach a point where it is hard to make the production code run faster, and that level of speed might still be a problem. What you could realize at this point is that just because you are doing UI testing, a testing technique like injection might still work. Often when people do UI testing, they do it in the context of the entire system, but just as you are accustomed to inject test doubles in unit tests, you can also do that in UI tests. When we do UI production code, there probably is a tendency to forget good programming principles like low coupling and high cohesion. But you should remember those principles – and if you forgot them, refactor your code.

Regarding the absence of brittleness, there are two major issues to address. Good unit tests are written in a way such that they depend very little on the environment in where they are running, ideally the only resource they use is memory and no disk, no network, etc. Again, this is more difficult to achieve for UI tests. The bottom line is that all kinds of tests should run in a context where their dependencies are controlled such that the outcome of the test is the same for all runs. For a unit test that might be nothing but memory, but for a UI test it may be a docker container or an entire separate virtual machine! At this point there often is a trade-off between speed and the absence of brittleness; running a UI test on the development machine may be faster, but may also make it more brittle.

The second issue to address regarding the absence of brittleness is asynchronicity. In contrast to a typical function call in a unit test, a UI/GUI interface often has an asynchronous nature. Thus after having interacted with the UI, the test code needs to wait for the result of the interaction to complete. One poor strategy that is often used is to expect that it happens immediately. That might work for some time, but suddenly when a UI test is run in a different environment it starts to fail. Another poor strategy (which is often used when you see the first one failing!) is to insert a single sleep statement. It might apparently fix your test right now, but eventually it will fail again – unless the sleep is very high, and you really cannot afford that. A better strategy is to sleep in a loop and repeatedly wait for the interaction to complete. That requires that you can really detect when the interaction is complete – and if you cannot, you should consider modifying the production code such that is becomes possible. Another consequence of this strategy is that depending on the sleep in the loop you either poll for the condition aggressively thereby slowing down the entire system, or sleep for a relatively long period; in both cases your test will run at a slower speed. The best strategy is to have the production code make a callback to the test code, for example in the form of an event.

Having mentioned the benefit of having the production code offering callbacks in the form of events leads to another suggestion. Consider building an object model into your production code and expose this object model to clients. This object model might either partly be the UI layer or exist entirely just below the UI layer. Having such an object model augmented with events for determining when the system reaches a specific state enable tests to run fairly fast and avoid the presence of brittleness.

I have seen many attempts to write UI and especially GUI tests. Almost all attempts have failed in the sense that tests are slow and brittle – with the result that they are really expensive to maintain. The first common mistake is to start writing such test against production code that was not written with this requirement in mind. It is just as if you are trying to unit test a class that was not written with unit tests in mind; you will have to refactor the class in order to be successful. The same holds for UI/GUI tests, but often the refactoring does not happen, because it is considered too expensive. Also, the developers capable of refactoring the production code are busy with more “important” tasks, the task of writing UI/GUI tests are assigned to less experienced people that to some degree are reluctant of refactoring the production code, thus resulting in poor tests being written. A variant of this is to let testers write UI/GUI tests – in my opinion that is a huge mistake, since writing such tests (including refactoring the production code as you go along) is probably one of the most difficult development tasks! When we build software we have a bunch of skilled people including architects, UX designers, testers and developers – and you probably would not let developers do the architecture, UX design and test design!