We live in a time of several revolutions/shifts in testing approach: in 90 till mid-2000 most testing activities were concentrated on desktop applications and most tools were designed to fit this purpose. Next period could be called web era, when more and more tasks could be performed online, generating demand for skills for web app testers and automation tools for web-based solutions.
With the parallel growth of the mobile application industry and the number of Android/iOS-based devices, testing skills and automation tools have become in demand. What all these “eras” have in common is the absence of success recipe due to lack of experience and de-facto industry standard. Something like that still can be observed when you have to work on very specific task where there is no known way of doing it right.
One of the ongoing revolutions in QA is testing of smart device-based solution, that will force QA specialists to adopt new skills, technologies and change the way how the test process is built.
To understand how “smart” the world around us is we can compare the number of MCUs per person manufactured each year. According to last years IEEE report, there are 20 B MCUs produced annually, which is about 3 per person. The number of smart devices connected to the internet almost achieved 7 B, which is close to 1 device per person; this number is expected to grow to 30 B by 2020, creating demand for tools, approaches and skilled QA specialists.
What is IoT testing and can it be automated?
If someone in the industry still is not convinced that longterm QA automation will help reduce costs, the era of smart electronics will definitely change their opinion.
Although there are multiple diagrams describing smart device network architecture, let’s focus on the narrow example of a smart meteostation that sends temperature and humidity data to remote storage and to display warnings sent to metsation. If we split it into modules and data processing layers, such a system could look like this:
If we split it into layers, it will be obvious that a smart device-based solution requires a different approach since it has to be tested on hardware, firmware and software level:
Hardware tests are typically performed using the bed of nails approach, and in some cases, manual quality control and smoke tests (literally – smoke tests). In most cases, it is easier to adapt an existing platform or outsource its creation to third-party and concentrate on FW and SW creation.
FW creation can also be done in different ways that could be divided by hardware abstraction level. But in this article, we will concentrate on testing SW, which includes simulating behaviour of a smart device network.
Although simulator can never fully emulate real-life scenarios, it is not uncommon to use them even in critical industries such as healthcare (e.g. EEG and ECG baseline signal generators). The same approach could be used for smart device testing as well: code should simulate the baseline device and baseline behaviour for all the device models supported by the platform. Sure, that does not test HW and FW, but it is cheaper and allows to add support of devices even before they hit market simply agreeing on message format.
Before choosing simulator architecture and functionality, teams should outline challenges that are seen in smart device testing:
- Interactivity tests: simulated device should talk both ways and will reply to command in a matter of milliseconds, that cannot be achieved by manual tests
- Variety of models: with the predicted growth of the smart device market to 30 B devices, it is safe to assume that there will be new manufacturers, new device types, new functions of devices. manual tests of each model for each release are not wise, so Simulator should be able to emulate all supported devices.
- Non-functional tests: getting data from a device and putting it to DB does not sound like a rocket since, but it becomes more complicated when the number of connected devices increased. Meaning, the simulation should be scalable
There are multiple mocking, simulation solutions available, but after a brief analysis of available tools, it obvious that as a long-term solution it would be easier to create own tool, specifically designed for needs of the project. First steps could be hard, but later adding a new device to simulator should be a matter of minutes if it requires just adding new device data and metadata templates:
As mentioned before, Simulator does not emulate FW and HW modules, so this should be taken into account building process for smart device solution QA.
Another possible limitation is that proposed simulator architecture is not designed to emulate any outages on FW or HW side, as well as data corruption during transmission and some other negative scenarios. This leaves less than 100 % availability, but at as the first step to QA process automation, it is wise to prioritise work on positive scenarios.
Use of simulator in the process together with automated data tests allows a team to:
- Make sure that SW part supports data processing from all smart device models
- Automate data processing tests since raw data generated by the simulator is available for automation scripts and can be compared against data processing result
- The simulator could emulate millions of devices
- Adding support of new model is a matter of minutes.
Written by IT Professional, Rama R Anem