Testing The Untestable

A long time problem for automated end-to-end testing has been testing production systems that have no test system available for testing.

A classic example is payment systems and other financial endpoints. Other examples I have worked with include college admission systems, insurance company APIs, loan system endpoints and many local and federal government services that cannot afford or are unable to provide test systems you can use.

Traditionally end-to-end testing with Selenium did not provide simple ways to overcome this problem. This in turn, frequently leads to automation that is very limited in scope and/or require clunky manual intervention and/or manual testing. These challenges only grow when we try to run our test suites in CI systems which require full automation.

Modern approaches have now given us tools and knowledge on how to address this. A couple of solutions are shown below

1. Create a local server that serves the same responses that the actual API will provide and then configure test runs to start and use this server. This typically means getting a sample response from the real API and then building test responses using this. One item that frequently needs to be addressed is using some form of unique ID generator to provide unique ID's that real responses would provide.
2. Use a framework such as Cypress or Playwright that provide you with the easy ability to mock out responses from a real server. These tools provide many of the pieces that you would end up building yourself if you went with the first option of a test server.

An important part of a good solution will be to test at multiple levels. Given the above solutions involve mocking out live responses you should consider integration tests (if possible) to verify that production endpoints are still valid and of course a base of high coverage unit tests for actual functionality. You might also consider a few happy path smoke tests for production with sad cases covered in test systems. Production monitoring of live endpoints rather than going through brower UIs is another option to consider.