IoT testing: how to become more flexible

Even before IoT testing, the automation and repeated execution of tests have been integral aspects of our day-to-day work as software developers. Continuous delivery and DevOps would otherwise be unimaginable. What are the advantages of test automation? Development teams receive speedy feedback on modifications to software code. They can also spend less time on manual testing. As a result, developers can truly streamline development in the name of agile processes.

Illustration that shows what happens when the IoT meets agile software development and DevOps.

With its billions of connected devices and continuous flows of data, the IoT has substantial non-functional requirements. But frameworks such as agile software development or DevOps also need different approaches to testing: teams must choose appropriate tools for their unique test scenarios and automate testing whenever feasible.

Developers have traditionally focused on automating functional tests. But as the Internet of Things (IoT) becomes more important, non-functional tests are also gaining in significance . Considering that billions of devices are interconnected, software must not only perform as expected, but also do so reliably under heavy load . I now want to explore the greatest challenges in this regard.

The top 3 challenges of IoT testing

The Internet of Things consists of billions of interconnected devices. This generates tremendous amounts of data but also places heavy loads on key IoT components. How can IoT testing automation be available on demand, while scaling to meet ever-growing demands on test infrastructure and IoT tooling requirements?

Challenge 1: Development teams must be able to integrate non-functional tests into their continuous tool chains, without needing to administer the requisite test environments themselves.

If load testing is to be as realistic as possible, it has to simulate the behavior of tens of thousands – if not hundreds of thousands – of sensors. Clearly, using such quantities of real sensors is essentially out of the question. This makes it necessary to virtually generate an appropriate number of sensors using a device simulator or similar test tool. Such pilot projects require a lot of hard work from development teams, which can only rarely commit enough resources. Before actual testing can even begin, teams must consider the personnel costs. The necessary infrastructure also costs money. What’s more, the teams have to procure and integrate appropriate systems into the network. Although such effort and expenditure is key to software quality, it also slows down development considerably. Due to this, teams might postpone tests to late stages of a project – or even cancel them altogether, which is much riskier.

Illustration that shows that non-functional testing entails considerable investments of time and money.

Non-functional testing requires tools and infrastructure that entail considerable investments of time and money before the very first test can even be run. In addition, the infrastructure must scale up or down to meet the latest requirements or test scenarios.

Challenge 2: It is impossible to predict which tools will be used to test a certain software artifact. For this reason, integrating new tools such as device simulators has to be a speedy and simple process.

Test infrastructure is not the only thing that must be versatile. Indeed, the range of potential test tools is truly diverse: from device simulators developed in-house to generic load generators, there is a suitable test tool for practically any pilot project.

Challenge 3: IoT testing environments and test systems are not static. Teams must be able to test their systems in and from various networks and at different geographic locations.

Even though IoT software platforms are available globally by definition, and despite agile development processes touting potentially shippable product increments, new software artifacts are often tested in internal networks. Instead, it should be as convenient as possible to switch between different test environments.

How to take a more flexible approach to IoT testing

Some aspects of the challenges above have already been mastered in practice. First, there are many providers of Infrastructure as a Service (IaaS) such as Amazon or Microsoft, which allow companies to rent computing power at the push of a button and make it available just as quickly. Second, nearly any software program can be installed automatically using configuration management (CM) products such as Chef or Puppet. Regarding a system under test (SUT), the two aforementioned partial solutions must be combined to make an easy-to-use solution.

Motivated by the challenges of the IoT and agile processes, the Bosch Software Innovations test center created a generic tool for fully automated testing. All Bosch divisions can use this tool to quickly and easily integrate testing tasks into their own development processes. The methods and approaches for this generic tool are the outcome of quality-assurance measures executed during development of the Bosch IoT Suite and Bosch projects for IoT applications:

Illustration that shows how the AUTOMATED TESTING SERVICE works.

The AUTOMATED TESTING SERVICE serves as the one and only touchpoint for test automation; it also coordinates the entire process and collects test artifacts. The tester simply initiates execution of testing and then waits for the results.

Step 0: The tester makes a REST request to initiate execution.

Step 1: The AUTOMATED TESTING SERVICE automatically requests the necessary virtual machines in the IaaS.

Step 2: The CM system rolls out the test configuration, consisting of simulators or test tools, to the virtual machines created in step 1. Testing then starts.

Step 3: The test artifacts are automatically transmitted from the IaaS to the AUTOMATED TESTING SERVICE.

Step 4: The AUTOMATED TESTING SERVICE automatically releases all computing power that it no longer needs.

Step 5: Via a REST request, teams can now retrieve the test artifacts locally from the AUTOMATED TESTING SERVICE and evaluate test results.

Wrap-up and outlook

Thanks to the integration of IaaS providers, teams no longer have to maintain their own test systems. What’s more, they pay only for the computing power that they actually use. And REST-based execution facilitates integration into existing toolchains. IaaS and the CM product make IoT testing considerably more scalable and versatile . It is easy to add more machines or other tools simply by modifying the REST request.

Illustration that shows the benefits of the AUTOMATED TESTING SERVICE.

The AUTOMATED TESTING SERVICE helps you significantly reduce your investments of time and money in test infrastructure. You pay only for what you actually use.

So, what’s next? Dr. Jürgen Kraus, system engineer at Bosch Software Innovations, summarizes: “We know that, as we dive even deeper into the world of things, more challenges lie ahead. By relying on the approaches presented here, however, we feel confident about tackling and mastering these challenges.”

What has been your experience with conducting IoT tests?

 

About the author

Felix Elliger

Felix Elliger

While studying IT Systems Engineering at Hasso Plattner Insitute in Potsdam, I started working as a research engineer in 2009. After finishing my master’s degree in 2011, I became a software developer, which ultimately served as a springboard for me to pursue my interest in testing in 2012. As a systems engineer, I gained expertise in test automation and its integration into agile processes. My focus today is on making developers’ lives easier by providing smart solutions for testing and test automation.