Want Better IoT Software Testing? Test in the Real World

The struggle is real for software testers, particularly in the IoT space.

At a time when the IoT market is expanding rapidly — Bain & Company predicts it will be worth $520 billion by 2021 — the pace of development has never been faster while the pressure to get it right, and in production, is also intense. That means testing, always a potential stumbling block in any development environment, is under more scrutiny than ever in the IoT space today.

Simply put, IoT software testing is difficult, said Arthur Hicken, evangelist with software testing tools maker Parasoft. “IoT represents a double challenge to testers,” Hicken said. “Embedded software is always tricky to test … and then you have safety (concerns), hardware that has to perform in real time, memory constraints, and it’s often all done in (the programming language) C. And you’re not just testing the device itself; you have to stand up the entire ecosystem and test that, so there’s a big chance a lot of things won’t be fully tested.”

The solution, Hicken suggested, is for IoT testers to approach the problem the way a hardware designer would. “Hardware companies in general have tended to behave better (when it comes to testing),” he said. “They think more about testing because they’re testing critical things like cars or planes. They look at the bigger picture.”

It’s time for IoT testers to, literally, go outside. “They have to go and test in the real world,” said Jeffrey Hammond, vice president and principal analyst at market research firm Forrester Research. Without a real-world vetting, even the best IoT project can fall apart, he said, and pointed to a company that put a beacon near a bridge not understanding that every time the bridge went up communication would stop completely.

The Case for Testing Early

Early IoT software testing “in the wild” would no doubt have picked up on the bridge problem quickly, which is why experts argue the sooner an IoT project is tested out in the real world, the better. In a Forrester report, “IoT Delivery Best Practice: Perform Real-World Testing,” Hammond and his colleagues explained how early testing in the place a device will be used can result in vital feedback that’s in time to make changes to the software, the hardware and even the customer’s expectations. And prompt testing of a product in its destined location will quickly spotlight challenges around installation, deployment or updating.

While it’s possible to make the case that all IoT software testing should be done early and on site, the potential of IoT devices to generate insane amounts of data makes the case for that type of testing even stronger. Real-world testing will show external problems — like electromagnetic interference — but it will also make it far easier to see and understand the impact of data transmission, the Forrester Report argued. Testers and developers will be able to see how networks slow down or crash, the impact on performance and the potential costs involved. Find this out early and decisions to put computing power on the edge versus the cloud become much easier to make and may not require a project get scrapped.