https://www.iotworldtoday.com/wp-content/themes/ioti_child/assets/images/logo/mobile-logo.png
  • Home
  • News
    • Back
    • Roundups
  • Strategy
  • Special Reports
  • Business Resources
    • Back
    • Webinars
    • White Papers
    • Industry Perspectives
    • Featured Vendors
  • Other Content
    • Back
    • Q&As
    • Case Studies
    • Features
    • How-to
    • Opinion
    • Podcasts
    • Strategic Partners
    • Latest videos
  • More
    • Back
    • About Us
    • Contact
    • Advertise
    • Editorial Submissions
  • Events
    • Back
    • Embedded IoT World (Part of DesignCon) 2022
Iot World Today
  • NEWSLETTER
  • Home
  • News
    • Back
    • Roundups
  • Strategy
  • Special Reports
  • Business Resources
    • Back
    • Webinars
    • White Papers
    • Industry Perspectives
    • Featured Vendors
  • Other Content
    • Back
    • Q&As
    • Case Studies
    • Features
    • How-to
    • Opinion
    • Podcasts
    • Strategic Partners
    • Latest videos
  • More
    • Back
    • About Us
    • Contact
    • Advertise
    • Editorial Submissions
  • Events
    • Back
    • Embedded IoT World (Part of DesignCon) 2022
  • newsletter
  • IIoT
  • Cities
  • Energy
  • Homes/Buildings
  • Transportation/Logistics
  • Connected Health Care
  • Retail
  • AI
  • Metaverse
  • Development
  • Security
ioti.com

Embedded Computing


Emerson

IoT Edge Data Modeling Favors Speed Over Monolithic Solutions

How data analysts should exploit edge data models for real-time data insight but also enlist the cloud for less time-sensitive tasks.
  • Written by Lauren Horwitz
  • 9th September 2020

Key takeaways from this article include the following:

  • Edge computing offers new opportunities for data analysis and modeling in real time.
  • IoT edge data models, however, require less monolithics approaches to data lakes and modeling.
  • Data analysts need new skills to be able to properly categorize, ingest and then manage this data at the edge and in in the cloud.

Enterprises are eager to use IoT data to understand costs, operations and their future prospects.

The data generated by mobile devices and Internet of Things devices (IoT) enables organizations to reduce costs, improve operational efficiency and innovate. But only if organizations can capture meaning from data at the right time.

Providing meaning and context to data, in turn, makes data analysts responsible for building data models that can deliver that meaning. Further, that data is high volume and coming at great speed, from many disparate locations.

In turn, data analysts should design edge data stores for speed and real-time insight in addition to monolithic data stores, like data lakes, to answer numerous questions for every business unit and purpose.

“Instead of flowing the data from domains into a centrally owned data lake or platform, domains need to host and serve their domain datasets in an easily consumable way,” wrote Zhamak Dehghani in a piece on moving from centralized data lakes for real-time data needs.

This is particularly true at the edge, where compute processing resources may be at a premium, even as the number of devices proliferates. With edge computing, a data algorithm can run on a local server or gateway, or even on the device itself, enabling more efficient real-time applications that are critical to companies.

“Edge puts that compute power right there, where the action is happening,” said Lerry Wilson, senior director innovation and digital ecosystems at Splunk, a machine learning platform.

A new forecast from International Data Corporation estimates that there will be 41.6 billion IoT devices, or “things,” generating 79.4 zettabytes (ZB) of data in 2025. This data also needs to be aggregated from myriad devices and business units — often siloed and in different formats, which creates additional complexity.

“The complexity inside these organizations has gone beyond the capacity of just keeping these devices up and running,” Wilson said. “You have to be able to see how this device interacts with this device. “From a security perspective, it’s absolutely critical — but also from an operational and business perspective.”

 

Data Volume and Velocity of Dictates New Edge Data Models

To effectively use edge computing for rapid data processing requires a different approach to data modeling and ingestion, practitioners emphasize. Data analysts need to consider how the data will be used and how much latency can be tolerated as well as its security and storage requirements.

That will require a hybrid approach to data ingestion as well as new edge data models that favor speed rather than overarching, monolithic data models.

“Taking the raw data from hundreds of thousands of sensors and doing minimal processing there and then sending the rest of the data to a centralized location — that approach is going to become more popular,” said Dan Sullivan, principal engineer at Peak Technologies.

So, for example, if IoT data can be used to detect a business-critical anomaly, that data may need to be processed right away. Environmental data over time, however, may not be critical and could be sent to the cloud or an on-premises data center for further processing instead.

At the same time, while decision makers want this data at the ready — fast-data preparation and cleansing can get in the way of fast data processing.

Indeed, nearly 40% of data professionals spend more than 20 hours per week accessing, blending and preparing data rather than performing actual analysis, according to a survey conducted by TMMData and the Digital Analytics Association.

IoT data still requires this preparation, but analysts say that it is now preferable to try to create less monolithic models upfront that require heavy logic data joining. The goal is to ingest data quickly, then query the data in targeted ways.

“Rather than figure out a complex solution, let’s have two simple solutions — rather than one monolithic solution. You don’t try to get one data model to answer all of your questions when they are really different. Because there is so much data, you want to target a subset of the data quickly.”

Sullivan noted that some of the data might also be sent to the cloud for machine learning and algorithm training. That data might not require such low latency or quick turnaround.

“The raw data is important from a machine learning perspective,” Sullivan said. “It doesn’t have the latency requirements that, say, anomaly detection data does. You may fork off that data for machine learning purposes and slowly load that up to a cloud object store.”

This new approach to data modeling echoes a paper on IoT data management, asserting that IoT data models need to be more flexible to accommodate real-time data management and large volumes of data.

“Database models that depart from the relational model in favor of a more flexible database structure are gaining considerable popularity, although it has been shown that parallel relational [database management systems (DBMSs)] outperform unstructured DBMS paradigms.”

 

IoT Data Management Skills Remain a Hurdle

These kinds of decisions about where the data goes and how to make the greatest use of it require unique skill sets, experts agree.

“Data scientists are … master of all jacks,” wrote Rashi Deshai in an article about the top skills for data scientists. “They have to know math, statistics, programming, data management [and] visualization … 80% of the work goes into preparing the data for processing in an industry setting.” WIth data volumes continuing to explode, data management has become a data scientist skill.

Organizations may lack these skills in-house, however. Either they need to cultivate these skills through additional education and training or enlist third-party partnerships.

“The holdup is going to be with people,” Sullivan said. “There just aren’t a lot of people who understand how to work with data at scale. It’s expensive to do this on-prem, but it requires a lot of engineering experience. People who can write the data-engineering pipelines are needed to extract value from it.”

As another data management indicates, this requires a jack-of-all-trades skill set.

The data echoes a long-standing skills shortage in IT data management.

According to IoT World Today’s August 2020 IoT Adoption Survey, lack of in-house skills is a blocker for IoT projects: 27% of survey respondents said that their lack of expertise was getting in the way of IoT deployments.

Splunk’s Wilson says that silos between operational and information technology departments have also been a holdup, but OT is beginning to see the virtues of automation and machine learning at the edge.

“People are starting to put the tools and the processes in place for real-time machine learning,” Wilson said. “They are embracing the concept — and the work to get there.”

 

 

 

Tags: Developers Embedded Computing

Related


  • Digital Pets Company
    Lonely in the Metaverse? Breed and Train Your AI-Driven Virtual Dog
    Pets autonomously learn and interact with owners – like in real life
  • Musk Offers to Buy Twitter for $43B
    The Tesla and SpaceX founder recently took a 9% stake in Twitter
  • Embedded IoT World 2022: Suzanne Deffree Talks DesignCon History
    Informa's group event director Suzanne Deffree sits down with Chuck Martin at this year's event.
  • Embedded IoT World 2022: Omdia's Josh Builta Talks Trends, Chips, 5G and More
    Omdia's head of AI and IoT kicked off Embedded IoT World this year. He sat down with Chuck Martin to talk about new trends, the chip shortage, 5G and more

Leave a comment Cancel reply

-or-

Log in with your IoT World Today account

Alternatively, post a comment by completing the form below:

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Content

  • Case Study: Developing a Custom Remote Sensing Solution for Nuclear Power Plants
  • Nvidia’s Arm Deal Hit With Fresh Regulatory Concerns in the UK
  • Collibra Raises $250M for Data Intelligence Cloud
  • IoT World Silicon Valley: Optimism for Connected Technology

Roundups

View all

IoT Product Roundup: PTC, Nokia, Arm and More

19th May 2022

IoT Deals, Partnerships Roundup: Intel, Nauto, Helium and more

14th May 2022

IoT Product Roundup: Amazon, Synaptics, Urban Control and More

27th April 2022

White Papers

View all

The Role of Manufacturing Technology in Continuous Improvement Ebook

6th April 2022

IIoT Platform Trends for Manufacturing in 2022

6th April 2022

Latest Videos

View all
Dylan Kennedy of EMQ

Embedded IoT World 2022: Dylan Kennedy of EMQ

Dylan Kennedy, EMQ’s VP of global operations, sat down with Chuck Martin at Embedded IoT World 2022.

Embedded IoT World 2022: Omdia’s Sang Oh Talks Vehicle Chip Shortage

Omdia’s automotive semiconductor analyst sits down with Chuck Martin at this year’s event

E-books

View all

How Remote Access Helps Enterprises Improve IT Service and Employee Satisfaction

12th January 2022

An Integrated Approach to IoT Security

6th November 2020

Webinars

View all

Rethinking the Database in the IoT Era

18th May 2022

Jumpstarting Industrial IoT solutions with an edge data management platform

12th May 2022

AI led Digital Transformation of Manufacturing: Time is NOW

9th December 2021

Special Reports

View all

Omdia’s Smart Home Market Dynamics Report

7th January 2022

Cybersecurity Protection Increasingly Depends on Machine Learning

28th October 2020

IoT Security Best Practices for Industry and Enterprise

20th October 2020

Twitter

IoTWorldToday, IoTWorldSeries

Europe’s First Automated Gas Station Convenience Store dlvr.it/SR45J9 https://t.co/eDJDJ7CxkI

25th May 2022
IoTWorldToday, IoTWorldSeries

Zero-Emission, Autonomous Cargo Ship Under Development dlvr.it/SR3xgb https://t.co/3oU7CrKWkW

25th May 2022
IoTWorldToday, IoTWorldSeries

Hannover Messe 2022: 5 Key Themes at This Year’s Show dlvr.it/SR3wtW https://t.co/j0fQUiU2LW

25th May 2022
IoTWorldToday, IoTWorldSeries

Can AI Help Stop Mass Shootings? dlvr.it/SR3tqK https://t.co/VxPBdajWA3

25th May 2022
IoTWorldToday, IoTWorldSeries

China has revealed an autonomous marine drone carrier dlvr.it/SR3qXs https://t.co/yMiiqeMNrZ

25th May 2022
IoTWorldToday, IoTWorldSeries

ISQ’s #UAV inspection drones will be on display at this year’s @hannover_messe dlvr.it/SR35dg https://t.co/U6QOQtMbGw

25th May 2022
IoTWorldToday, IoTWorldSeries

👀 Looking to integrate #AI into your manufacturing process? Explore how #manufacturers can begin the process of… twitter.com/i/web/status/1…

25th May 2022
IoTWorldToday, IoTWorldSeries

The U.S. Army is getting a 5G boost for #AR #VR capabilities from #5G network provider @OceusNetworks.… twitter.com/i/web/status/1…

24th May 2022

Newsletter

Sign up for IoT World Today newsletters: vertical industry coverage on Tuesdays and horizontal tech coverage on Thursdays.

Special Reports

Our Special Reports take an in-depth look at key topics within the IoT space. Download our latest reports.

Business Resources

Find the latest white papers and other resources from selected vendors.

Media Kit and Advertising

Want to reach our audience? Access our media kit.

DISCOVER MORE FROM INFORMA TECH

  • IoT World Series
  • Channel Futures
  • RISC-V
  • Dark Reading
  • ITPro Today
  • Web Hosting Talk

WORKING WITH US

  • Contact
  • About Us
  • Advertise
  • Login/Register

FOLLOW IoT World Today ON SOCIAL

  • Privacy
  • CCPA: “Do Not Sell My Data”
  • Cookies Policy
  • Terms
Copyright © 2022 Informa PLC. Informa PLC is registered in England and Wales with company number 8860726 whose registered and Head office is 5 Howick Place, London, SW1P 1WG.
This website uses cookies, including third party ones, to allow for analysis of how people use our website in order to improve your experience and our services. By continuing to use our website, you agree to the use of such cookies. Click here for more information on our Cookie Policy and Privacy Policy.
X