Even before IoT testing, the automation and repeated execution of tests have been integral aspects of our day-to-day work as software developers. Continuous delivery and DevOps would otherwise be unimaginable. What are the advantages of test automation? Development teams receive speedy feedback on modifications to software code. They can also spend less time on manual testing. As a result, developers can truly streamline development in the name of agile processes.
Developers have traditionally focused on automating functional tests. But as the Internet of Things (IoT) becomes more important, non-functional tests are also gaining in significance. Considering that billions of devices are interconnected, software must not only perform as expected, but also do so reliably under heavy load. I now want to explore the greatest challenges in this regard.
The top 3 challenges of IoT testing
The Internet of Things consists of billions of interconnected devices. This generates tremendous amounts of data but also places heavy loads on key IoT components. How can IoT testing automation be available on demand, while scaling to meet ever-growing demands on test infrastructure and IoT tooling requirements?
Challenge 1: Development teams must be able to integrate non-functional tests into their continuous tool chains, without needing to administer the requisite test environments themselves.
If load testing is to be as realistic as possible, it has to simulate the behavior of tens of thousands – if not hundreds of thousands – of sensors. Clearly, using such quantities of real sensors is essentially out of the question. This makes it necessary to virtually generate an appropriate number of sensors using a device simulator or similar test tool. Such pilot projects require a lot of hard work from development teams, which can only rarely commit enough resources. Before actual testing can even begin, teams must consider the personnel costs. The necessary infrastructure also costs money. What’s more, the teams have to procure and integrate appropriate systems into the network. Although such effort and expenditure is key to software quality, it also slows down development considerably. Due to this, teams might postpone tests to late stages of a project – or even cancel them altogether, which is much riskier.
Challenge 2: It is impossible to predict which tools will be used to test a certain software artifact. For this reason, integrating new tools such as device simulators has to be a speedy and simple process.
Test infrastructure is not the only thing that must be versatile. Indeed, the range of potential test tools is truly diverse: from device simulators developed in-house to generic load generators, there is a suitable test tool for practically any pilot project.
Challenge 3: IoT testing environments and test systems are not static. Teams must be able to test their systems in and from various networks and at different geographic locations.
Even though IoT software platforms are available globally by definition, and despite agile development processes touting potentially shippable product increments, new software artifacts are often tested in internal networks. Instead, it should be as convenient as possible to switch between different test environments.
Some aspects of the challenges above have already been mastered in practice. First, there are many providers of Infrastructure as a Service (IaaS) such as Amazon or Microsoft, which allow companies to rent computing power at the push of a button and make it available just as quickly. Second, nearly any software program can be installed automatically using configuration management (CM) products such as Chef or Puppet. Regarding a system under test (SUT), the two aforementioned partial solutions must be combined to make an easy-to-use solution.
Motivated by the challenges of the IoT and agile processes, the Bosch.IO test center created a generic tool for fully automated testing. All Bosch divisions can use this tool to quickly and easily integrate testing tasks into their own development processes. The methods and approaches for this generic tool are the outcome of quality-assurance measures executed during development of the Bosch IoT Suite and Bosch projects for IoT applications:
Step 0: The tester makes a REST request to initiate execution.
Step 1: The AUTOMATED TESTING SERVICE automatically requests the necessary virtual machines in the IaaS.
Step 2: The CM system rolls out the test configuration, consisting of simulators or test tools, to the virtual machines created in step 1. Testing then starts.
Step 3: The test artifacts are automatically transmitted from the IaaS to the AUTOMATED TESTING SERVICE.
Step 4: The AUTOMATED TESTING SERVICE automatically releases all computing power that it no longer needs.
Step 5: Via a REST request, teams can now retrieve the test artifacts locally from the AUTOMATED TESTING SERVICE and evaluate test results.
Wrap-up and outlook
Thanks to the integration of IaaS providers, teams no longer have to maintain their own test systems. What’s more, they pay only for the computing power that they actually use. And REST-based execution facilitates integration into existing toolchains. IaaS and the CM product make IoT testing considerably more scalable and versatile. It is easy to add more machines or other tools simply by modifying the REST request.