With the huge increase in mobile phone usage, mobile testing is quickly becoming one of the most vital skills a tester can have.
How often do you use your phone?
If you are anything like me then the answer will range from ‘a lot’ to ‘too much’, which is perhaps an answer that is indicative of the huge change occurring in how people use web apps, with generations of users now accustomed to having the internet available to them 24/7 on their smartphone.
With people moving away from accessing the web on their desktop or laptop devices, companies releasing products on the web need to think about how their site looks and works on a mobile phone first, and a computer second.
Today’s users will not be sat at a computer, they will be reaching into their pocket and loading up the website on their smartphone. Therefore, mobile testing is quickly becoming one of the most vital skills a tester can have. In the mobile world, we have two options to access a product. We either use a browser on the phone to access a website, or we use an app.
However, it’s not as simple as that. Firstly, there’s the operating system of the phone. Android is the most popular, with iOS close behind. Windows is next up after iOS and Android, but there is a huge step down in usage rates. Beyond OS, there are also two types of app; native and web wrapper (often called hybrid).
Web wrapper basically means that the app is showing you the exact content that is available to you from the website, but ‘wrapping’ it in such a way to make it appear as an app. For some websites this won’t be that obvious, but for the most part it’s really obvious when you realise what to look for.
Native apps on the other hand are completely separate from websites; all of the functionality you see within a native app is coming from the app itself – no web stuff involved. This means that, on the whole, native apps beat web wrapper apps on basically all fronts – they’re faster, easier to use, look better and are ultimately more fit for purpose.
Native apps are developed to be used exclusively on phones, so you’re going to have a much better time using them. Think of all of the best apps you use; Facebook, Snapchat, WhatsApp, YouTube, Twitter, Spotify, basically all banking apps… they’re all native for a reason!
Web wrappers usually use websites which aren’t developed and tested with app usage in mind, which results in a lot of UI problems. Furthermore, automation tools struggle because they will have to use the web’s HTML DOM to control the app – but more on this later.
It’s not really a surprise that mobile developer’s talk about web wrappers with such distaste – it’s basically the cheap option if you already have a website. However, it is a widely used solution by a lot of big companies as they have spent 10-15 years building up a web-based product, and have huge web teams.
The move to mobile happened quickly, and as such mobile developers are very much in demand right now. As far as browser testing goes on mobiles, it’s very similar to browser testing on a computer. There are still nuances which you will only find when using mobile devices, for example screen size or landscape mode.
Mobile testing considerations
Here is a non-exhaustive list of things to look out for while mobile testing that you likely (most of the time) won’t be able to try using a computer:
- Screen size testing: mobile testing includes tablets or iPads, so screen size can range from the 4-inch diagonal of the iPhone 5 all the way up to almost 13-inches diagonal for the newest iPads. Apps have to work across all the widely used screen sizes
- Internet connectivity testing: people don’t always have 4G or WiFi connection, so we have to either use tools to ‘throttle’ the connection, or actually physically go somewhere with a poor signal
- Usability testing: mobile phones these days have colour-blind modes and other various usability options; we have to make sure our apps work properly with these settings
- Interrupt testing: what happens to our app when you receive a text or email whilst using it? If the user soft-closes the app (not fully ending the apps activity), many things can happen. Our app needs to be able to deal with these interruptions
- Upgrade testing: arguably one of the most important aspects of mobile testing, users are required to update their app if they want to have access to the latest functionality, from the Play Store for Android and the App Store for iOS. Apps often need to retain data (for example login information or preferences) after an update, which is a big difference to web development as the updating process is done server-side rather than device-side
- OS integration testing: quite often apps will come up with some functionality, which OS’s will pick up in later versions. This will often render the functionality obsolete in the app, and will need to be removed. A good example is screenshots – some apps would let you fiddle about with the screenshot within the app, but now the latest Android and iOS OS’s have that built in, which means the apps need to remove it as its superfluous.
These are only six areas of mobile testing which differ from testing from a computer, there are still more out there; and the list is only growing as our phones get smarter and smarter.
Mobile testing – what to test?
So, we now know that mobile testing offers a lot of complexity and interesting scenarios you might not have initially considered. So how do we decide what to test? We know we can’t test everything… it’s never an option. We have to come up with a plan.
The most obvious way to decide what we test is thinking about the use cases – what are the most widely used devices? Thankfully due to all the data collection these days, we know pretty accurately which versions of OS and which phones people are using, right down to a week- by-week basis.
Using this data, we can decide in the team whether we care about the five most used phones, or the 10 or 15 most used… This gives us a pretty good start, but when you realise you are probably going to have to actually purchase these phones to physically test the apps, your budget really starts to feel the strain.
Bear in mind that new phones are released frequently, and that people love buying the newest model, and you face a potentially never-ending spending spree on your test devices.
This is where options such as device farms come in handy, for example Sauce Labs or Firebase Test Lab. These third-party companies offer access to remote devices for you to test your apps on, but again it is down to pricing and whether it is cost effective to choose these options.
Let’s talk about testing tools
Lots of the tools you are probably familiar with, such as New Relic and Splunk, are naturally usable for mobile apps, helping to provide some useful information. In one mobile team I worked in, we used these tools to track how many users were using certain versions of our apps, and what device they were using them from.
Furthermore, we could see which paths the users were taking through the apps, and used this information to inform our testing strategy – there is little point testing a scenario which users never actually use. Aside from surveillance tools like those, there are other tools that are used during mobile testing which are external to the app’s code.
Charles is an example of one such tool, which is used to set up a proxy on the phone’s network. Once this is in place, we can see all of the network traffic going through the phone. This allows us to potentially mess around with the way that the app works, and we can simulate a lot of situations that real users will see. This is obviously preferable to walking around in the middle of a forest, trying to simulate a bad internet connection!
For all of you Selenium enthusiasts, Appium is one tool that is branched off from Selenium and uses JSONWire Protocol to send commands to the phone. For the most part the syntax is exactly the same, but there are obviously some slight differences.
For example, there is a thing called ‘contexts’ in apps which use some web stuff and some native stuff (the hybrid web wrappers mentioned earlier). Because the native code lives in a native DOM, and the web code lives in the HTML DOM, we have to switch between the two contexts to be able to interact with the objects that live in each one.
There is even the potential for there to be multiple web contexts. This is why native apps are far, far nicer to automate. That, and it’s a lot easier to have control over how the elements are named when you are working on a native app – something all you Selenium aficionados will understand is important; using ID’s to find elements which you can ask developers to set is perfect. Using other methods… not so much.
Learn from past experiences
The last thing I’d like to speak about are a couple of experiences I have had during my time working on mobile apps teams, in the hope that I can pass something on to someone who might end up in a similar situation to myself.
A lot of the time, the little things are what makes apps fun and cool to use. Everybody likes finding little ‘Easter eggs’ the app developers leave behind, and often some of these little funky features are actually quite useful. One of the apps I was working on required a calendar to pop up and the user would pick a number from the calendar view and be shown the information for that day.
However, the functionality for when the user picks a day which hasn’t happened yet (and as such there is no page to navigate to), would be for nothing to happen at all.
Now, if you think about that as a user you will realise that it’s not immediately obvious why nothing has happened. The user won’t know that the app has actually received the touch event, and might get confused as to why the app ‘isn’t working’.
I had a little think about what I would expect the app to do if I was doing something I shouldn’t be, and decided that if the calendar number did a small ‘wiggle’ from side to side that it would inform the user they’re doing something incorrect. After checking with the developer, he found some existing library which could do it, and voila, done in five minutes. No product person needed, and the app has become nicer to use.
Often, when working on a development team, the developers will be working on the same product. That is obviously the case in mobile development too, but when you are making the same app across two different operating systems, you can actually think of them as separate products to some extent. The team I worked on had some Android developers, and some iOS (Windows wasn’t something we supported).
On a project where we were up against it, both sets of developers were working from the same set of designs for a native app. However, not all of the desired functionality could work at every part of the project. Sometimes the designs had to change to allow for this.
However, when these designs changed, this change wasn’t always communicated to the opposite OS developers. This meant that we had two products which were meant to be heading towards the same end result, but in actuality we had two apps which were straying further and further away from each other.
I think the only way this could have been avoided is through my involvement; ideally, I would have been so involved with the developers’ process that I could see what was happening and how far astray we were getting. Sadly, because of the time constraints, that wasn’t something I was able to do – and we ended up suffering because of it.
So, I hope this article has shed some light on mobile testing for those of you who are keen to become further involved in it. I honestly believe mobile development is going to expand and develop even more than it has done in the past couple of years – as soon as we have batteries that last more than a day – who will need laptops!
Rob Catton, consultant, Infinity Works