Cucumber or Gauge? Things to consider when choosing the best Test Automation Framework for your team
What are the different types of software tests? Which ones can be automated and what are the best automation tools? These questions made me wonder if we still need so many tests, from classics like unit testing to modern approaches like End-to-End tests for UI. In today’s world any software test can be automated but it’s not easy to develop all these different types of tests unless you use the proper tools and automation strategies.
Unit Testing has always been the key verification stage of a successful software test. With Test-Driven Development (TDD), developers start writing tests even before or simultaneous with the writing of the actual code. But even if the units perform as designed, that doesn’t necessarily mean the provided service functions as desired. That’s when teams come up with Integration Tests to make sure the communication among different parts of the architecture works as expected. It makes more sense with Service-Oriented Architecture and Microservices. Some teams also do Functional tests to add one more layer of verification to test the functionality of each service. Just like Unit Tests, Functional tests may also benefit from the concept of Mocking.
End-to-End tests
All projects, including frontend projects providing the GUI to users, need to be tested, of course. You can add all kinds of needed tests to a UI project. But usually when it comes to UI tests it’s more about End-to-End (e-2-e) tests involving test scenarios from the user’s perspective. We don’t want to redo unit tests or any kind of service tests in UI/e-2-e tests. Below is the Agile Test Automation Pyramid and the distribution of tests within it.
What is Automated Testing? SmartBear, 2019
Behaviour-Driven Development (BDD) is a very popular way to develop e-2-e tests. It follows the concepts of TDD along with the concepts of Domain-Driven Design (DDD), which makes it an easy process for emulating user interactions with UI.
How to choose the best Automation Framework?
There is a long list of available Testing Automation Frameworks, from open-source to commercial. The best option may vary based on the needs of different teams. The key points to consider are:
- Who is going to develop the e-2-e tests? QA Engineers or Software Developers?
- What needs to be tested?
- What is the technical culture of the team?
- What programming language(s) will be used along with the scripting language?
For example, Robot Framework is one of the best available options when it comes to QA Engineers. It’s a keyword-based framework and easy to use for non-developers. But customizing the keywords and adding more keywords may lead to unexpected complexities.
Karate is a good choice for testing APIs and web-services and just recently introduced some experimental and unstable UI testing features. Galen is a nice option when it comes to User Experience and checking the appearance of the product in different screens. But again, this is not what you need for e-2-e tests.
Cucumber is still the most popular scripting tool and is widely used in BDD development and automated tests along with Selenium, which provides the browser automation so the project can emulate user interactions with Web-UI. If the team-culture is more about classic approaches like the Gherkin style of scripting - with its well-known keywords “Given” for setting things up, “When” for the testing section, and “Then” for verifying the test result - then Cucumber is the best choice for the team.
And more coding is needed, of course, to create a customized framework behind the test scripts. That is how behaviors and steps are defined to be used in the test scripts. Each team has its own preferred programming language(s). Some Frameworks are language-specific, like Watir, which is a good choice for Ruby teams. So, when choosing an Automation Framework, it matters what languages are supported.
What Gauge is and why it’s a game changer
Gauge is a young automation framework with the potential to become the standard in test automation in the near future. It covers all the BDD Concepts other competitors follow and offers a plethora of fascinating features as below:
Shared Data
With Gauge you can store/retrieve run-time data much like a static container or a Session object.
Passing data from File
Gauge is not just limited to inline tables. You can do Data-Driven testing by passing data files in formats like txt and csv.
Reporting
Gauge has its own reporting feature. You may still prefer to use reporting tools like Serenity BDD, but it’s still a cool feature.
Parallel Execution
You can turn on this feature by setting an environment variable and you can also define the number of streams. It will provide you a multi-threading feature for running tests in parallel to finish testing quickly if your tests can be run in parallel.
Environments
This might not be a big deal for modern development frameworks with built-in configuration features like “properties” files. But Gauge has its own environment management solution and files. You can put Gauge “properties” files in the Gauge defined project structure and then address the environment name while running the tests. You can handle it with different profiles as well but it’s an additional feature that some other frameworks lack.
Documentation
Gauge has nice documentation and code snippets, and a responsive community. It is still not as popular as Cucumber, and has fewer contributors, but this is to be expected from a young framework.
Reusable Scripts
Gauge offers two scripting levels: Specifications and Concepts. Concepts may be used to provide reusable logic to Specification-level scripts.
Execution Levels
You may address a Specification level, Scenario level, directory level and more levels while running the tests. It can be handled in different ways, of course, but it’s nice when the framework lets you do this with no effort.
Tear Down Step
With Gauge you can define Teardown steps to be run after the scenario.
More Hooks
Gauge offers many hooks so you can simply use the proper annotation to address when you want a piece of code to be run. Like @BeforeSpec, @AfterSpec, @BeforeScenario, @AfterScenario and more...
Rerun Failed Tests
You can force Gauge to rerun the failed tests.
Continue on Failures
While the default behavior is breaking execution on the first failure in a step, you can mark the steps as “ContinueOnFailure”.
Works with multiple languages
Currently, Gauge supports Javascript, Java, C#, Python and Ruby.
Some of the mentioned features are available in some frameworks or with plugins. But Gauge offers an attractive set of features with no need for any other plugins.
And one more thing to consider...
The Scripting Syntax of Gauge is different from the well-known Gherkin style. Gauge follows Markdown Syntax for scripting the tests in an unstructured way - which means more liberty to make the test like a specific document and also more risk of ending up with scripted tests in many different styles. Having some in-team standards will help to prevent any chaos.
Finally, here is a brief comparison of the growth-rates of Cucumber and Gauge in the past 12 months:
Gauge appears to be getting more attention. So, I suggest you consider it as an option if it supports your programming language.
And they tested happily ever after...
More readings and resources
19 Best Automated DevOps Testing Tools For 2019, BOJANA DOBRAN, phoenixnap.com, 2018
9 top open-source testing automation frameworks: How to choose, Joe Colantonio, techbeacon.com
Why I chose Gauge over Cucumber, Thivya lakshmi, medium.com, 2018
Gauge vs Cucumber, Abhishek Singh, github.com
