Real-Time Location Systems: A Moving Target for IoT
Data Streams, Data Lakes
When IoT location data is collected, it is seldom as a steady stream. As IoT begins to fulfil the promise of thousands or even millions of devices in the field, incoming data spikes and overflows remain a concern.
Skerrett said the industry has experienced an increase in integrations that use software like MQTT along with Apache Kafka, yet another message system. He said the two complement each other.
Kafka is among a host of open source big data tool standards that have arisen in recent years to handle processing in real time. Along with associated streaming and query tooling, Kafka has become a nexus for new event processing methods for IoT location tracking and other purposes.
Most enterprises today don’t use the data their machines and sensors generate continuously, because they cannot access and process it,” said Kai Waehner, who is a field engineer and architect at Confluent, a chief vendor driving Kafka software methods today. The data goes unused either because of technical limitations or complexity, which makes it difficult implement with legacy technologies, he said.
Waehner said inaccessible, unprocessed data hinders developers from being able to clearly communicate the business value of IoT data.
Microservices such as Kafka can be distributed on a large scale, he said. Thus, they can be “integrated with all required systems and allow processing of the data in real time.” The biggest underlying technical challenge for many is integrating with several different technologies, standards and interfaces, he continued.
As with MQTT, there are plenty of alternatives to Kafka components, especially when used to create “data lakes” that hold fast arriving, undifferentiated IoT data before it is sorted and processed. Competitive tools to Kafka in all its varieties include Apache Spark, RabbitMQ and other systems.
Build vs. Buy
Advanced skills required for integrating different elements in today’s big data systems cause some in-house IoT teams to emphasize “buy” over “build.” That can mean opting for more encompassing, end-to-end IoT platforms.
Such commercial systems often include assorted open source big data components under the hood.
“When you think about IoT scale, of the ingestion of streams of data into a data lake for real-time processing, it is a nontrivial task,” according to Steven Glapa, vice president of product management at Aeris Communications.
“Companies with a strong manufacturing bent, for example, may not put Kafka pieces together as deftly as companies that have already been doing it for a number of years,” Glapa said.
Today, the pressure on shippers for faster decision making renders traditional methods of integration unacceptable, and pre-integrated systems can be a quicker fit with back-end systems already in place, according to Luis Parajes, executive vice president of sales and marketing at collaborative logistics platform vendor Turvo Inc.
“We organize every component of the logistics network so that our customers have a 360-degree view of their supply chain and how it’s performing,” he said. The goal is to give people real-time visibility into assets, items and orders in the supply chain along with the execution of transportation and settlement activities, he indicated.