Want Better IoT Software Testing? Test in the Real World

Tester as Guardian

Moving testing out into the real world will also mean that the role of tester must change, Hammond said. Testers will be on hand to see exactly what works, and what doesn’t, and will perhaps for the first time have an insight into customer needs and expectations that’s greater than the developer’s. With that is going to come pressure, Hammond warned. “This is the time to be skeptical about the use case,” he said. “You need to make sure that the use case is providing real value.” IoT testers should have the opportunity to weigh in on every part of the process, he said, including things like how updates are going to be addressed.

And it won’t stop there, Hicken added. The complex nature of IoT is going to put pressure on testers to become more technical because otherwise they simply won’t be able to do the job, he said. IoT testers will need to be able to at least read code and have a deep bench of usability and user experience skills. These so-called hybrid testers are going to be the backbone of IoT moving forward. “In this environment having a hybrid tester is super critical,” he said.

Setting Expectations and Standards

But even the best hybrid testers can’t catch every single error, Hammond said. Companies should test in the real world, and upskill their testers, but they should also be prepared for the process to be time-consuming, costly and not bulletproof, he said. His best advice: Build in a “debug mode” that will allow testers to capture anomalies over a period of time, as that is sometimes the only way to ensure thorough testing.

There may also be some light at the end of the tunnel from standards organizations finally ready to consider software, Hicken said. Certain industries like aerospace and health care have had robust standards and expectations around testing for years, but largely on the hardware side, he said. What IoT software development needs is more standards bodies outlining the scope of software testing. And he thinks UL2900 might be the right place to begin. “I think UL has made electrical devices easy to test in an objective manner,” he said. “Software gets a lot trickier. But the organization has been trying to craft a standard and is going through a revision to tighten it and make it cleaner. It’s a challenge — how do you know what’s in a device and how do you apply appropriate tests?”

The standards bodies aren’t there yet, Hicken said, but it is coming. “What you’re seeing is that software is being put into a more constrained box that is very normal for what the hardware people are used to. We’re now applying those principles to software from development to engineering but we’re not there yet,” he said. “We’re still waiting for agreement on a set of standards.”