10 Killer Applications of the IoT and Augmented Reality

While one of the central promises of the Internet of Things is to make our world smarter, that can only happen when the right information is provided at the right time.

Enter augmented reality (AR) technology, which can be used to visualize data from hundreds of sensors simultaneously, overlaying relevant and actionable information over your environment through a headset. “With AR, the immersive user experience comes to life,” says Zeki S. Gunay, CEO of sensor firm Cratus Technology Inc. (San Jose). For instance, if you are operating heavy machinery, you can get a real-time view of which components are in need of replacing. If you a farmer, you could see at a glance what the moisture level is of your fields, while monitoring the fuel levels of your tractor. Gunay foresees nearly limitless applications for AR technology across industries

Don DeLoach, CEO of Infobright, is also optimistic about the convergence of AR and the IoT: “I think this is a really cool upcoming wave that will be associated with IoT,” he says. “If you think about it, the vast repositories of data will enable an AR lens into the scenarios in ways that provide near immediate insight at a level of depth unimaginable previously.”

Notable organizations—like Microsoft, NASA, Autodesk, Volvo, and Caterpillar—are putting their weight behind AR. Microsoft, in particular, has proclaimed that its HoloLens technology is suited for a broad range of use cases and it has announced that companies ranging from Autodesk to NASA to Volvo are testing its Hololens augmented reality platform.

In this feature, we share 10 ways this technology could be used to address a range of real-world problems:

1. Bringing City Administration into the 21st Century

While cities like Boston are using Moneyball-like tactics to rank how well it is serving its 656,000 residents, such data-driven approaches are rare for city administration. But as smart city initiatives gain in favor, managing a city could become as focused on data as it is on the whims of city officials and their constituents. As that begins to happen, augmented reality could be useful to help sort through the mountains of data that smart cities collect. “If you think about all the data points you get from IoT, it is easy to envision building up a historical repository of data including things like street traffic, pedestrian traffic, weather conditions, traffic incident data, building data, and much more,” says Infobright’s Don DeLoach. “All of this data can then feed augmented reality apps for allowing you to envision and ‘engage’ in hypothetical situations,” he says.

This technology could be used by emergency responders. “Moreover, those same first responders might plug certain variables into an incident as it is unfolding to ‘see’ the prediction of what will happen. They could visualize where the crowds will go, how the flood will expand, where the fire might move and which people and/or facilities would be impacted,” DeLoach says. The technology could also enable first responders to practice how they respond to challenging situations such as interacting with hazardous materials. “This would allow them to manage their response much more effectively as a result—likely saving lives,” DeLoach says.

Augmented reality technology will also be of interest to architects and city planners who could create virtual cityscapes or transportation grids and then demonstrate them to their clients—a vast improvement over 2-D blueprints.

On a related note, the nation-state of Singapore has invested tens of millions of dollars into a virtual version of its city—dubbed “Virtual Singapore.” City planners have said that they plan to use the virtual replica for virtual test-bedding to model and simulate crowd dispersion to prepare for an emergency.

Cratus is working with a structural monitoring company to develop an AR-based structural integrity sensing system that it believes will become a standard for civil infrastructure applications. “This system will be a cornerstone component of smart cities by delivering structural integrity data to AR headsets,” Gunay says. The company is using wireless technology by Fujitsu Components for data delivery to the AR headset.“In the next generation system, the user experience will get even more immersive by the addition of our proprietary mesh algorithm.”

2. Cutting Inefficiency and Diagnosing Technical Problems 

Augmented reality technology can increase the efficiency and productivity of industrial facilities by enabling staff to see the most pertinent sensor data in a dashboard-like view, says Zeki S. Gunay, CEO of Cratus Technology Inc., a sensor-centric IoT product development company. “We are connecting as many sensors as we can together, extracting data from them, and, depending on the context, displaying what is the most user-friendly, most efficient, and most applicable,” he says. Its BlueBrain platform is capable of collecting data from hundreds of combinations of sensors and transmit that data in real time to augmented reality or virtual reality headsets.

The company is agnostic regarding the choice of headset but thinks that transparent glasses that offer a heads-up-display offer the most potential. “We want to make sure that the person sees his or her environment as they normally see it. But with the headset, they can extract the data without speaking to the device or using an interface,” Gunay says. “You could be on a manufacturing floor, and you are walking down the aisle that is populated with machines and you can look at one machine and you can see its temperature. You can also see the vibration data, efficiency data, and whatever else is needed.”

The technology could also be used to guide workers on the shop floor. 

3. Can a Helmet Protect Workers in More Ways Than One?

The Los Angeles–based firm DAQRI has developed a helmet that can just protect workers from falling objects but enable them to spot hazards and help them from making mistakes when repairing equipment. The smart helmet can provide employees with guided work instruction and thermal vision to spot overheating components. The technology can reduce errors and repair time while doing away with the need to consult printed instructions. The DAQRI device is also capable of doing diagnostics and detecting hazards using thermal vision.

4. Maintaining a Fleet of Heavy Equipment Vehicles

Heavy machinery maker Caterpillar is using augmented reality technology for predictive maintenance enabling a user to look at a machine and instantly see a visual overlay that states when various components need to be replaced, how much fuel has been used, and how much weight, say, a backhoe is carrying. The technology could also actively assist users in maintenance operations, walking them through the steps required to, for instance, change an air filter.

5. Monitoring Crops and Farm Vehicles

With droughts becoming more common globally, farmers can’t afford to waste water. But sensors in the soil can help inform them exactly when watering is necessary. Augmented reality technology can help farmers not only monitor moisture, but they can also track the temperature of their crops and keep track of their equipment. “A farmer could be walking through the field and see the moisture level of the soil, the temperature of the crops. AR can also be used to monitor the performance of your water pump or your harvester to see if it is operating normally or running out of gas or if the temperature is too high,” says Zeki S. Gunay, CEO of Cratus Technology, Inc.

6. Detecting and Preventing Pump Failures

Flowserve, a manufacturer and aftermarket service provider of flow control products and services, is aiming to simplify maintenance of its centrifugal pumps with mixed reality. Real-time sensors will monitor machine conditions, for anomalies and a real-time CFD analysis triggered to pinpoint the cause. A virtual image—basically a CAD model—is projected onto the machine and when combined with animation will show the exact steps for making the fix.

Earlier this year, CIO reported that Flowserve hooked up with the AI company SparkCognition to predict pump failures five days in advance. SparkCognition is itself looking to augmented reality to help share the management analytics and predictions formulated by its software.

7. Making Technical Documentation Actually Useful

User manuals and technical documentation are notoriously boring. Making matters worse, such documents can sometimes over-complicate straightforward procedures—whether it be diagnosing a problem with electronics or assembling a piece of industrial equipment or Ikea furniture.

Augmented reality can help with this by overlaying instructions and contextual information directly over parts that are ready to be assembled.

The company Bosch has partnered with AR on what they call a “Common Augmented Reality Platform,” which can be used to overlay text, circuit diagrams, videos, and augmented 3-D animations over a piece of equipment.

On a related note, Bosch Rexroth has developed technology—known as ActiveAssist and ActiveCockpit—to help guide workers on exactly how to assemble components, breaking down each step that is necessary. Although not strictly an AR technology, ActiveAssist and ActiveCockpit are similar in theme in that they help visualize data to help humans assembly custom components more quickly and with fewer errors. 

8. Cutting the Cost of Shipping Freight

Shipping goods across oceans would be safer, less expensive, and less polluting if it was managed with augmented reality, according to Rolls-Royce. The company has a vision of replacing human-controlled cargo vessels with drone ships controlled at remote control centers. Rolls-Royce envisions a future where a single group of professionals—using AR technology—would be able to monitor hundreds of drone shifts that otherwise rely on AI and machine learning to navigate.

9. Visualizing Product Design

Imagine if designers could see the products they are developing explode to life in the form of a life-sized virtual replica that can appear like it is sitting in the same room. The designers could instantaneously make changes to the designs in Autodesk software, changing the color of, say, a motorcycle and tweaking the shape of the body, as Microsoft demonstrated last year. And because of the Microsoft HoloLens ability to interact with the real world, designs can also be overlaid over existing vehicles. The ability to instantaneously iterate could cut down on the need for creating prototypes.

Augmented reality technology also could open up a new world of collaboration between product designers and engineers who could interactively discuss product refinements. Already, NASA is using mixed reality technology (which seamlessly merges the real and virtual.) to replicate the surface of Mars and companies like VW and Volvo are using the technology as a complement to traditional CAD/CAM design. 

The following video from Autodesk describing its Fusion product innovation platform showcases a variety of augmented reality use cases.

10. Making Medical Education Interactive

Heads up display technology by itself could revolutionize education requiring hands-on training. A surgeon, for instance, could guide interns through a procedure using a first-person view. Or a surgeon in the midst of a difficult procedure could consult an expert located outside of the operating room, and share a real-time view of the surgery. In fact, both use case has already been proven using the first generation of Google Glass. Despite the fact that the first iteration of Glass failed to spark consumer interest, the technology has already inspired many startups to develop AR apps for the technology. Philips also has worked with Accenture to develop surgical applications of Glass. 

There are also similar apps that use tablet technology to display a patient's internal anatomy when the device is positioned over the body. 

Microsoft believes that its HoloLens technology will open up new avenues in education. The company is partnering with Case Western University and the Cleveland Clinic. “Using 3-D images, students can explore, alter, and examine subjects like anatomy from every angle,” says a Microsoft spokesperson.