Software testing smart devices
What are the key challenges QA teams face when software testing smart devices? What is the best way to approach testing them and how do you overcome the challenges they present?
The concept of smart devices is not a new one, but the rate at which they have advanced over recent years is staggering, particularly if you consider the rise of the smartphone. Research by Ofcom found that in 2008 only 17% of people in the UK owned a smartphone. Over a decade later and the figure has risen to more than 78%, even going as high as 95% amongst 16 to 24-year-olds.
Include into the mix myriad other devices, such as smartwatches, cars, speakers, refrigerators, heating systems and other general home automation, and it’s easy to see how some QA teams can be left feeling overwhelmed.
Even if all these devices operated independently, it would still be a task for testing teams. The fact these wide-ranging devices, with all their different functions and outputs, now integrate into the wider world of IoT, means you have a plethora of factors to consider when testing.
So, what are the key challenges QA teams face when trying to test smart devices? What’s the best way to approach software testing smart devices, and what are the solutions to taming ‘tech that travels’?
Software testing smartphones
Smartphones are everywhere. Though there are obvious market leaders, each brand has a huge range of handsets, each with differing features, software and OS functions. Testing smartphones is a long and varied process with many challenges, leading to calls for hyper-specialised testing devices for mobile (Delemaro et al, 2006). However, there are some common challenges that all testers have to consider.
Firstly, battery life. No matter how innovative a product is, if it doesn’t have power, it’s useless. Testers need to ensure the battery doesn’t interfere with the phone’s other functions – for example, low power mode can cause a phone to automatically turn off some functions of the device, potentially leading to unreliable testing, particularly for installation and memory testing. Testers should ensure the device is tested with a variety of charges and in all states of power use.
The sheer variety of OS across mobile devices means testers have to account for variance when running specific items. For example, Android is known to have a very peculiar input source (Amalfitano et al, 2011) compared to other devices, which can affect functional testing when automated. Testers must ensure they account for the different OS when selecting what tests to run.
For many the ‘smart’ part of smartphones is the quality of their applications. As well as variety in models and OS, there are many different kinds of apps and all need to be tested due to their interaction and effect on each other. Some apps are native to that specific device, and so can be more predictable, whereas HTML5 or hybrid apps work across devices, and so can be less stable.
Testers need to be aware of the differences and factor that into their case management.
What if you could have your computer on your wrist? A concept that has been on the ‘cool list’ since James Bond had every kid pretending their calculator watch could fire lasers and drive a car. Smartwatches are perhaps achieving what the jet pack couldn’t – they are allowing us to live in the future, today.
Apple was, perhaps, the first to make the smartwatch truly ‘work’ in the mainstream and the other device manufacturers have since followed suit, meaning there are a number of great items on the market today. However, smartwatches come with a number of challenges.
Just like a smartphone, if it doesn’t have power, it’s useless. Most users wear their devices 24/7, especially to record things such as exercise and sleep. Therefore, they have to be economical on power, as unlike with smartphones, long charging sessions are not an option. Testers will need to ensure the device display only turns on when a notification is pushed, or the user raises their hand in a ‘checking’ direction. The tester should also put the watch under heavy use, to check all the functions work at various stages of charge, and through various levels of activity.
The sensors on a device are key and have to account for a wide variety of user-states (sleep, active, sedentary etc), therefore it can be difficult for testers to isolate how well the device sensors are interpreting data – particularly when it often relies on how quickly it can correspond to a connected smartphone. Speed is key: check how quickly the data is shared between a device and the watch, and use this as the foundation of your testing.
Very few smartwatches are independent, the majority need to be able to connect to several other devices. If a watch has trouble syncing or connecting to another device, it could be having trouble with its network protocol, which is now a vital part of device usability (Jiang, Chen et al 2015). Testers also need to test connectivity from different locations, ranges, power status and in states of charge/use.
Apps & OS
As well as the running of the device itself, testers need to test how the OS is performing and how it interacts with different types of apps. How does the mobile network operator function with the device or what about the scripting conflicts of each app? A tester also needs to consider how they will test such variances, for example, will they use several real devices, or run them on an emulator?
The latter is known to miss errors otherwise picked up on real devices, but can save on initial significant cost (Selvam 2011). A variety of testing methods, including functional and usability testing, is usually the best way to ensure the device is up to scratch.
Internet of Things
From home automation to transport and industry, the Internet of Things (IoT) is growing quickly. So much so, that Gartner has predicted that by 2020, there will be 20.4 billion IoT devices connected to the Internet. However, there are still a number of pitfalls that testers need to be wary of.
Software can be installed to protect devices like a laptop or PC, but for IoT devices, there isn’t much support. Many devices have default password and usernames that the average user never changes, meaning they are vulnerable to intrusion.
Testers also need to factor in the wider effect that IoT devices can have on Internet security. Something like a botnet attack can use items such as video recorders or cameras to cause huge disruption to a server.
Issues such as data flow between devices, data encryption and, if automating testing, the integrity of testing software, are all important.
Sometimes testers must simply rely upon users to protect their networks. However, many users will look to save costs with cheaper routers which often contain fewer safety measures. Also, some users may simply not have the knowledge on how to protect their network at all.
When testing something like a website, it’s easier to run through the scenarios that most users will encounter as they scroll through their tablets, smartphones or laptops. However, it’s far harder to replicate a users’ environment with IoT devices.
Where will the device be used, at home or at work? How many other devices will be using the same network and will the device need to leave and rejoin it? Testers have to ensure they have a strong understanding of the product and get creative. They need to try and imagine as many different scenarios as possible and factor that into their testing, including how the device will work with others.
Another key challenge is a more obvious one – cost. Testing IoT devices can be expensive, especially when you factor in different environments and integrations. The testing takes a lot of time as testers are having to cover so many bases and may need special testing software. Many believe this is partly due to a lack of thorough testing of IoT apps, and this can mean that testing becomes very complex and that in turn increases cost.
Interaction with other devices
Compatibility testing is always a thorn in the side of the tester, but in the context of IoT, it can become even more frustrating. There can be seemingly endless interactions making it simply too difficult to analyse and test every device for every release in every environment.
The only realistic way for testers to address this task is to focus on the most popular operating systems and communication methods (such as Bluetooth), on the most popular devices, in the hope that their efforts have the widest reach, before moving on to more niche items.
Firmware is an integral part of many IoT devices, and with that comes the need for constant updates. For testers, every new update is a chance for a bug or error to slip through the net, with potentially dire consequences. If a missed bug gets through into a standard device, the issue can usually be simply tested and fixed before there is too much damage.
But if a bug gets through into an IoT device, such as a home automation system, it can have far wider-reaching ramifications.
Scott Sherwood, Founder, Testlodge