Take a Step Back - Mobile Testing

Mobile Testing

Mobile Testing introduces unique challenges to your test approach and your test tooling. Take a step back and ask WHY?

Most software projects today include a dedicated mobile app or a mobile compatible website (responsive design etc.). Therefore it is critical that your test strategy and approach is tailored for mobile testing. The attached mind map highlights some of the topics you need to consider for mobile testing but this is by no means exhaustive. At the time of writing this article, Apple had just launched iPhone 6S (with iOS 9), which included 3D Touch (formerly ForceTouch) and new device specific gestures such as Peek and Pop. This increases the user experience but further complicates the mobile testing landscape.

Mobile Challenge

"Here today, gone tomorrow" is probably the best way to characterise the pace of change in the mobile market. In fact, many of today's most popular handsets and tablets become out-dated and irrelevant in a matter of months. The mobile market is extremely dynamic, unpredictable, and fragmented. The numerous operating systems and plethora of platform versions, networks, hardware, and form factors make it challenging to maintain application quality.

New devices contain the latest or near-latest OS versions, and they usually automatically upgrade to the newest available version. There are no guarantees that an application developed according to an older version will function properly with a new version; enterprises have no choice but to conform to this pace and continually develop and test version updates for their applications. This is in contrast to the upgrade cycle for desktop operating systems. Windows 10 has been released recently, but many enterprises, never upgraded to Windows 8 and a large percentage still use Windows 7 and a small percentage some are still running Windows XP (released in August 2001).

Adapting the Software Development Lifecycle for Mobile

Mobile application testing simply cannot be served by the traditional development and QA cycle. A tremendous number of customers will instantly adopt newly released mobile devices and OS versions and connect right away to applications and websites. Although an organisation may not be prepared to introduce an updated application version, users expect nothing less than a flawless user experience.

In the mobile market, the risk accumulated between product releases is much greater than in traditional software. This is because the market changes are much faster. For example, when developing a traditional web application such as online banking, the chances that the IE browser version will change within a six-month development cycle is unlikely. In mobile, the leading devices and OS versions are very likely to change with this six-month timeframe. This leaves enterprises no choice but to accelerate the release cycle in order to limit risk exposure. Choosing not to quickly release updated versions will make a mobile app irrelevant.

Interoperability and Compatibility

The mobile world is characterised by a plethora of devices, screen resolutions, operating systems, network operators and locations. In a world comprised of tens of thousands of different client devices, and over 400 operators around the globe, testing your mobile application can become a real problem.

Interoperability is a completely new mobile phenomenon. Browsers do not receive phone calls but phones do. Similarly, device events such as SMS or notifications can occur at any moment, causing unexpected interruptions to a running application or transaction. For example, an online purchase can be interrupted by an incoming call or dropped because of connectivity issues. How will your mobile app interact with connected wearable devices such as Apple Watch and similar watches?

Traditionally software developers have the luxury of assuming all desktops are basically the same, regardless of manufacturer, CPU, and memory. In mobile, no such assumption can be made. The differences between devices are too great and cannot be ignored. Therefore, compatibility, also known as porting, is added to the development cycle to validate application and content compatibility, performance and user experience across devices.

Automation, Automation, Automation

To deliver the required short development lifecycle and to still meet the users expectations, the QA timeframe must be shortened, despite the added complexity of interoperability and compatibility. As development cycles become shorter over time, the need for automation in regression testing has turned from a "nice to have" into mandatory, particularly when using agile or iterative development methodologies. For agile methodologies to work correctly, we need to ensure that the already-developed components continue to work and that they are not exposed to defects that were introduced in the most recent sprint. The automation framework must be able to work across different devices and operating systems and support multiple versions of the application under test in parallel.

Contrary to some beliefs, automation is an enabler for success and not a cost-reduction exercise.

Mobile Performance

Specific attention should be given to the performance of the mobile application across supported devices. This includes monitoring of resource usage in active and idle states (memory, CPU, battery, data usage) and ensuring that for long running tasks that the user is well informed as to the progress. There are various tools available from the OS vendors and 3rd parties to monitor resource usage and detect memory leaks.

Memory: Ensure that memory usage in active and idle state corresponds with profile expectations. If memory usage increases over time, care should be taken to check for memory leaks.

CPU: Ensure that CPU usage in active and idle state as per expectations. Apps running in the background should have minimal to zero CPU usage. Specific care should be taken with CPU usage during reconnection logic and during synchronisation (where applicable).

Battery: Ensure that the application does not negatively affect battery usage, especially when running in background mode and idle state. Battery usage is highly correlated to CPU usage.

Data: Most mobile data plans have over time moved away from "all you can eat" unlimited data plans to capped or metered data plans. This means that specific care should be taken to ensure that only the data absolutely required is transferred by your application. Data should be compressed whenever possible. It is also recommended that data is coalesced or throttled on the server side to prevent any unnecessary or stale data being transferred to the mobile device.

Mobile Security

Security adds a new layer of complexity to testing mobile applications. With mobility, the client side of your applications is more important than ever. Instead of accessing your website from the safe confines of work or home, your customers and/or employees can now perform sensitive transactions anywhere. As a result, mobile applications, and the devices and data they interact with, can be more easily attacked.

Mobile application security is not just about the application located on the device; it is also about the network traffic going between that application and its back-end server, as well as the server components themselves.

This means not only reviewing these components from the outside while the application is running (dynamic testing), but also at the code level (static testing). Consider the following components of a mobile application and the different testing options:

Client

• Dynamic (client-side fuzzing)
• Static (binary and code review, filesystem/memory analysis)

Network

• Dynamic (traffic analysis)

Server

• Dynamic (full dynamic testing of back end)
• Static (full code review of back end code base)

Any mobile testing approach that doesn't involve testing the fully running application in its production-like configuration, with all its various components interacting with each other, is an exercise in theory rather than real-world security testing. This means that it is important to get full external end to end connectivity testing (WiFi, 3G/4G etc.) as early in the lifecycle as possible.

Mobile Device Lab

A lot of the mobile apps enterprises develop are targeted at staff which permits you to have some (if limited) control over the devices that you are required to support. For client facing mobile apps (uploaded to 3rd Party App Stores) you have much less control over the devices in use and thus you have a bigger testing challenge.

Even if you were to decide to build a fully stocked test lab of all potential devices your clients might use, you would probably need to replace approximately 10 devices each quarter to remain current. This would come at a very high cost in money and time.

There are 3 solutions to the device fragmentation problem.

1. Test the edge cases (boundaries). Test on the lowest spec devices (lowest CPU, screen resolution, memory, oldest OS, slowest network etc.) and repeat on the highest spec devices. Some high-level assumptions could be made that if it works on the worst and best, all devices in between should work.
2. Test the lowest common denominators. While the number of individual devices may be large, there is a lot of commonality between devices and they can be grouped into device families. Again high-level assumptions could be made that if it works on a Samsung Android 4 tablet that it should work on a Toshiba Android 4 tablet.
3. Use a shared test lab. Instead of building your own test lab, you could use a shared test lab provided by test specialist vendors or give the app to the vendor to test on your behalf.

In reality, a combination of all solutions will need to be taken. The number of devices that an application should be tested against should be ramped up as it moves through the SDLC. Development and low-level testing should be run against a small number of key popular devices (e.g. 4-6), regression should be executed against a slightly larger set (e.g. 10-12) and finally prior to launching a new application, acceptance testing should be completed against a large set of device families (e.g. 20-30).

So before define your test approach for your next mobile app project, take a step back…