https://www.iotworldtoday.com/wp-content/themes/ioti_child/assets/images/logo/footer-logo.png
  • Home
  • News
    • Back
    • IoT World 2020 News
  • Strategy
  • Special Reports
  • Galleries
  • Business Resources
    • Back
    • Webinars
    • White Papers
    • Industry Perspectives
    • Featured Vendors
  • Other Content
    • Back
    • IoT World 2020 News
    • Q&As
    • Case Studies
    • Features
    • How-to
    • Opinion
    • Video / Podcasts
  • More
    • Back
    • About Us
    • Contact
    • Advertise
    • Strategic Partners
  • IOT World Events
    • Back
    • Internet of Things World: San Jose
    • IoT World 2020 News
Iot World Today
  • NEWSLETTER
  • Home
  • News
    • Back
    • IoT World 2020 News
  • Strategy
  • Special Reports
  • Galleries
  • Business Resources
    • Back
    • Webinars
    • White Papers
    • Industry Perspectives
    • Featured Vendors
  • Other Content
    • Back
    • IoT World 2020 News
    • Q&As
    • Case Studies
    • Features
    • How-to
    • Opinion
    • Video / Podcasts
  • More
    • Back
    • About Us
    • Contact
    • Advertise
    • Strategic Partners
  • IOT World Events
    • Back
    • Internet of Things World: San Jose
    • IoT World 2020 News
  • newsletter
  • IIoT
  • Cities
  • Energy
  • Homes/Buildings
  • Transportation/Logistics
  • Connected Health Care
  • Retail
  • AI
  • Architecture
  • Engineering/Development
  • Security
ioti.com

Embedded Computing


Emerson

IoT Edge Data Modeling Favors Speed Over Monolithic Solutions

How data analysts should exploit edge data models for real-time data insight but also enlist the cloud for less time-sensitive tasks.
  • Written by Lauren Horwitz
  • 9th September 2020

Key takeaways from this article include the following:

  • Edge computing offers new opportunities for data analysis and modeling in real time.
  • IoT edge data models, however, require less monolithics approaches to data lakes and modeling.
  • Data analysts need new skills to be able to properly categorize, ingest and then manage this data at the edge and in in the cloud.

Enterprises are eager to use IoT data to understand costs, operations and their future prospects.

The data generated by mobile devices and Internet of Things devices (IoT) enables organizations to reduce costs, improve operational efficiency and innovate. But only if organizations can capture meaning from data at the right time.

Providing meaning and context to data, in turn, makes data analysts responsible for building data models that can deliver that meaning. Further, that data is high volume and coming at great speed, from many disparate locations.

In turn, data analysts should design edge data stores for speed and real-time insight in addition to monolithic data stores, like data lakes, to answer numerous questions for every business unit and purpose.

“Instead of flowing the data from domains into a centrally owned data lake or platform, domains need to host and serve their domain datasets in an easily consumable way,” wrote Zhamak Dehghani in a piece on moving from centralized data lakes for real-time data needs.

This is particularly true at the edge, where compute processing resources may be at a premium, even as the number of devices proliferates. With edge computing, a data algorithm can run on a local server or gateway, or even on the device itself, enabling more efficient real-time applications that are critical to companies.

“Edge puts that compute power right there, where the action is happening,” said Lerry Wilson, senior director innovation and digital ecosystems at Splunk, a machine learning platform.

A new forecast from International Data Corporation estimates that there will be 41.6 billion IoT devices, or “things,” generating 79.4 zettabytes (ZB) of data in 2025. This data also needs to be aggregated from myriad devices and business units — often siloed and in different formats, which creates additional complexity.

“The complexity inside these organizations has gone beyond the capacity of just keeping these devices up and running,” Wilson said. “You have to be able to see how this device interacts with this device. “From a security perspective, it’s absolutely critical — but also from an operational and business perspective.”

 

Data Volume and Velocity of Dictates New Edge Data Models

To effectively use edge computing for rapid data processing requires a different approach to data modeling and ingestion, practitioners emphasize. Data analysts need to consider how the data will be used and how much latency can be tolerated as well as its security and storage requirements.

That will require a hybrid approach to data ingestion as well as new edge data models that favor speed rather than overarching, monolithic data models.

“Taking the raw data from hundreds of thousands of sensors and doing minimal processing there and then sending the rest of the data to a centralized location — that approach is going to become more popular,” said Dan Sullivan, principal engineer at Peak Technologies.

So, for example, if IoT data can be used to detect a business-critical anomaly, that data may need to be processed right away. Environmental data over time, however, may not be critical and could be sent to the cloud or an on-premises data center for further processing instead.

At the same time, while decision makers want this data at the ready — fast-data preparation and cleansing can get in the way of fast data processing.

Indeed, nearly 40% of data professionals spend more than 20 hours per week accessing, blending and preparing data rather than performing actual analysis, according to a survey conducted by TMMData and the Digital Analytics Association.

IoT data still requires this preparation, but analysts say that it is now preferable to try to create less monolithic models upfront that require heavy logic data joining. The goal is to ingest data quickly, then query the data in targeted ways.

“Rather than figure out a complex solution, let’s have two simple solutions — rather than one monolithic solution. You don’t try to get one data model to answer all of your questions when they are really different. Because there is so much data, you want to target a subset of the data quickly.”

Sullivan noted that some of the data might also be sent to the cloud for machine learning and algorithm training. That data might not require such low latency or quick turnaround.

“The raw data is important from a machine learning perspective,” Sullivan said. “It doesn’t have the latency requirements that, say, anomaly detection data does. You may fork off that data for machine learning purposes and slowly load that up to a cloud object store.”

This new approach to data modeling echoes a paper on IoT data management, asserting that IoT data models need to be more flexible to accommodate real-time data management and large volumes of data.

“Database models that depart from the relational model in favor of a more flexible database structure are gaining considerable popularity, although it has been shown that parallel relational [database management systems (DBMSs)] outperform unstructured DBMS paradigms.”

 

IoT Data Management Skills Remain a Hurdle

These kinds of decisions about where the data goes and how to make the greatest use of it require unique skill sets, experts agree.

“Data scientists are … master of all jacks,” wrote Rashi Deshai in an article about the top skills for data scientists. “They have to know math, statistics, programming, data management [and] visualization … 80% of the work goes into preparing the data for processing in an industry setting.” WIth data volumes continuing to explode, data management has become a data scientist skill.

Organizations may lack these skills in-house, however. Either they need to cultivate these skills through additional education and training or enlist third-party partnerships.

“The holdup is going to be with people,” Sullivan said. “There just aren’t a lot of people who understand how to work with data at scale. It’s expensive to do this on-prem, but it requires a lot of engineering experience. People who can write the data-engineering pipelines are needed to extract value from it.”

As another data management indicates, this requires a jack-of-all-trades skill set.

The data echoes a long-standing skills shortage in IT data management.

According to IoT World Today’s August 2020 IoT Adoption Survey, lack of in-house skills is a blocker for IoT projects: 27% of survey respondents said that their lack of expertise was getting in the way of IoT deployments.

Splunk’s Wilson says that silos between operational and information technology departments have also been a holdup, but OT is beginning to see the virtues of automation and machine learning at the edge.

“People are starting to put the tools and the processes in place for real-time machine learning,” Wilson said. “They are embracing the concept — and the work to get there.”

 

 

 

Tags: Developers Embedded Computing

Related


  • Edge computing
    Emerging Edge Cloud Architecture Continues to Shake Out
    Edge cloud architecture is going to bring about new capabilities. But as data-intensive functionality comes together at the edge, technologies need to develop, then converge, first.
  • Training and Development on the Mechanism of Metal Gears. in the design of information related to business
    Turning to Rust Development For IoT Performance
    If you haven’t heard of it, Rust is an up-and-coming programming language that provides secure application performance for device-run code.
  • Training and Development on the Mechanism of Metal Gears. in the design of information related to business
    How to Run Docker on IoT Devices
    Here’s how to run Docker on IoT devices, which enables you to bring container-based app development to the edge.
  • Jetting to the Stars Using Containers for Development
    The Department of Defense develops for space travel, bombers and jets. It turned to containers for development to build and battle-test its mission-critical systems.

Leave a comment Cancel reply

-or-

Log in with your IoT World Today account

Alternatively, post a comment by completing the form below:

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Related Content

  • Argent Case Study
  • MES For Dummies
  • Super SIM: The Future of IoT
  • IoT in Utilities Market Brings Resilience in Wake of COVID-19 Pressure

News

View all

Private LTE Market Projected to Grow to $13 Billion

12th January 2021

IoT World Announces 2021 IoT World Advisory Board

9th December 2020

White Papers

View all

The eSIM Cookbook – Towards the Next Generation of Connected Devices

22nd February 2021

eSIM Delivers Greater Freedom for OEMs – by Beecham Research and Truphone

22nd February 2021

Special Reports

View all

Cybersecurity Protection Increasingly Depends on Machine Learning

28th October 2020

Webinars

View all

Weber’s Journey: How a Top Grill Maker Serves Up Connected Cooking

25th February 2021

From Insights to Action: Best Practices for Implementing Connected Device Security

15th December 2020

Galleries

View all

Top IoT Trends to Watch in 2020

26th January 2020

Five of the Most Promising Digital Health Technologies

14th January 2020

Industry Perspectives

View all

IoT Spending Holds Firm — Tempered by Dose of ‘IoT Pragmatism’

1st December 2020

The Great IoT Connectivity Lockdown

11th May 2020

Events

View all

IoT at the Edge

17th March 2021

Embedded IoT World 2021

28th April 2021 - 29th April 2021

IoT World 2021

2nd November 2021 - 4th November 2021

Twitter

IoTWorldToday, IoTWorldSeries

📢 Announcing #EIOTWORLD sponsor, @aicas_IoT — a flexible, more efficient approach to embedded realtime application… twitter.com/i/web/status/1…

4th March 2021
IoTWorldToday, IoTWorldSeries

Microsoft Ignite 2021: Innovation in COVID-19 Era Signals Future Trends dlvr.it/RtwYcg

4th March 2021
IoTWorldToday, IoTWorldSeries

At Microsoft Ignite: How IoT and Robotics Are Driving Industry 4.0 dlvr.it/Rttgwj

3rd March 2021
IoTWorldToday, IoTWorldSeries

🎙️ Introducing #EIOTWORLD speaker, Obinna Ilochonwu, Industrial IoT Architect at Schlumberger. 📅 Join his session… twitter.com/i/web/status/1…

2nd March 2021
IoTWorldToday, IoTWorldSeries

#Smartbuilding technology lays the foundation for #energyefficiency efforts but also new COVID-19 goals, such as… twitter.com/i/web/status/1…

2nd March 2021
IoTWorldToday, IoTWorldSeries

IoT Remote Monitoring Helps Enterprises Traverse COVID-19 and Beyond dlvr.it/RtZ3K5 https://t.co/owJXYf1gkO

26th February 2021
IoTWorldToday, IoTWorldSeries

Securing the Industrial Internet of Things dlvr.it/RtYfYk https://t.co/khUn79dvQD

26th February 2021
IoTWorldToday, IoTWorldSeries

📢 Announcing #EIOTWORLD sponsor, @BluetoothSIG — the global standard for simple, secure wireless connections. ➕ Le… twitter.com/i/web/status/1…

26th February 2021

Newsletter

Sign up for IoT World Today newsletters: vertical industry coverage on Tuesdays and horizontal tech coverage on Thursdays.

Special Reports

Our Special Reports take an in-depth look at key topics within the IoT space. Download our latest reports.

Business Resources

Find the latest white papers and other resources from selected vendors.

Media Kit and Advertising

Want to reach our audience? Access our media kit.

DISCOVER MORE FROM INFORMA TECH

  • IoT World Series
  • Channel Futures
  • RISC-V
  • Dark Reading
  • ITPro Today
  • Web Hosting Talk

WORKING WITH US

  • Contact
  • About Us
  • Advertise
  • Login/Register

FOLLOW IoT World Today ON SOCIAL

  • Privacy
  • CCPA: “Do Not Sell My Data”
  • Cookies Policy
  • Terms
Copyright © 2021 Informa PLC. Informa PLC is registered in England and Wales with company number 8860726 whose registered and Head office is 5 Howick Place, London, SW1P 1WG.
This website uses cookies, including third party ones, to allow for analysis of how people use our website in order to improve your experience and our services. By continuing to use our website, you agree to the use of such cookies. Click here for more information on our Cookie Policy and Privacy Policy.
X