From the Blogosphere
Difference Between #BigData and Internet of Things | @ThingsExpo #IoT #M2M
What does it mean, as a vendor, to say that you support the Internet of Things (IoT) from an analytics perspective?
By: William Schmarzo
Jun. 22, 2017 05:00 PM
A recent argument with folks whose intelligence I hold in high regard (like Tom, Brandon, Wei, Anil, etc.) got me thinking about the following question:
What does it mean, as a vendor, to say that you support the Internet of Things (IoT) from an analytics perspective?
I think the heart of that question really boils down to this:
What are the differences between big data (which is analyzing large amounts of mostly human-generated data to support longer-duration use cases such as predictive maintenance, capacity planning, customer 360 and revenue protection) and IoT (which is aggregating and compressing massive amounts of low latency / low duration / high volume machine-generated data coming from a wide variety of sensors to support real-time use cases such as operational optimization, real-time ad bidding, fraud detection, and security breach detection)?
I don’t believe that loading sensor data into a data lake and performing data science to create predictive analytic models qualifies as doing IoT analytics. To me, that’s just big data (and potentially REALLY BIG DATA with all that sensor data). In order for one to claim that they can deliver IoT analytic solutions requires big data (with data science and a data lake), but IoT analytics must also include:
If you cannot manage real-time streaming data and make real-time analytics and real-time decisions at the edge, then you are not doing IOT or IOT analytics, in my humble opinion. So what is required to support these IoT data management and analytic requirements?
The IoT “Analytics” Challenge
Consequently to support real-time edge analytics, we need to provide detailed data that can flag observations of concern, but then doesn’t overwhelm the ability to get meaningful data back to the core (data lake) for more broad-based, strategic analysis.
One way that we see organizations addressing the IoT analytics needs is via a 3-tier Analytics Architecture (see Figure 1).
Figure 1: IoT Analytics 3-Tier Architecture
We will use a wind turbine farm to help illustrate the 3-tier analytics architecture capabilities.
Tier 1 performs individual wind turbine real-time performance analysis and optimization. Tier 1 must manage (ingest and compress) real-time data streams coming off of multiple, heterogeneous sensors. Tier 1 analyzes the data, and processes the incoming data against static or dynamically updated analytic models (e.g., rules-based, decision trees) for immediate or near-immediate actions.
Purpose-built T1 edge gateways leverage real-time data compression techniques (e.g., see the article “timeseries storage and data compression” for more information on timeseries databases) to only send a subset of the critical data (e.g., data that has changed) back to T2 and T3 (core).
Let’s say that you are monitoring the temperatures of a compressor inside of a large industrial engine. Let’s say the average temperature of that compressor is 99 degrees, and only varies between 98 to 100 degrees within a 99% confidence level. Let’s also say the compressor is emitting the following temperature readings 10 times a second:
99, 99, 99, 98, 98, 99, 99, 98, 99, 99, 100, 99, 99, 99, 100, 99, 98, 99, 99…
You have 10,000 of readings that don’t vary from that range. So why send all of the readings (which from a transmission bandwidth perspective could be significant)? Instead, use a timeseries database to only send mean, medium, mode, variances, standard deviation and other statistical variables of the 10,000 readings instead of the individual 10,000 readings.
However, let’s say that all of a sudden we start getting readings outside the normal 99% confidence level:
99, 99, 99, 100, 100, 101, 101, 102, 102, 103, 104, 104, 105, …
Then we’d apply basic Change Data Capture (CDC) techniques to capture and transmit the subset of critical data to T2 and T3 (core).
Consequently, edge gateways leverage timeseries compression techniques to drive faster automated decisions while only sending a subset of critical data to the core for further analysis and action.
The Tier 1 analytics are likely being done via an on-premise analytics server or gateway (see Figure 2).
Figure 2: IoT Tier 1 Analytics
Tier 2 optimizes performance and predicts maintenance needs across the wind turbines in the same wind farm. Tier 2 requires a distributed dynamic content processing rule generation and execution analytics engine that integrates and analyzes data aggregated across the potentially heterogeneous wind turbines. Cohort analysis is typical in order to identify, validate and codify performance problems and opportunities across the cohort wind turbines. For example, in the wind farm, the Tier 2 analytics are responsible for real-time learning that can generate the optimal torque and position controls for the individual wind turbines. Tier 2 identifies and shares best practices across the wind turbines in the wind farm without having to be dependent upon the Tier 3 core analytics platform (see Figure 3).
Figure 3: Tier 2 Analytics: Optimizing Cohort Performance
Tier 3 is the data lake enabled core analytics platform. The tier 3 core analytics platform includes analytics engines, data sets and data management services (e.g., governance, metadata management, security, authentication) that enable access to the data (sensor data plus other internal and external data sources) and existing analytic models that supports data science analytic/predictive model development and refinement. Tier 3 aggregates the critical data across all wind farms and individual turbines, and combines the sensor data with external data sources which could include weather (humidity, temperatures, precipitation, air particles, etc.), electricity prices, wind turbine maintenance history, quality scores for the wind turbine manufacturers, and performance profiles of the wind turbine mechanics and technicians (see Figure 4).
Figure 4: Core Analytics for Analytic Model Development and Refinement
With the rapid increase in storage and processing power at the edges of the Internet of Things (for example, the Dell Edge Gateway 3000 Series), we will see more and more analytic capabilities being pushed to the edge.
How Do You Start Your IoT Journey
How effective is your organization at leveraging data and analytics to power your business (or operational) models?
We have tweaked the Big Data Business Model Maturity Index to help organizations not only understand where they sit on the maturity index with respect to the above question, but also to provide a roadmap for how organizations can advance up the maturity index to become more effective at leveraging the wealth of IOT data with advanced analytics to power their business and operational models (see Figure 5).
Figure 5: Big Data / IoT Business Model Maturity IndexMaturity Index
To drive meaningful business impact, you will need to begin with the business and not the technology:
If you want to monetize your IOT initiatives, follow those simple guidelines and you will dramatically increase the probability of your business and monetization success.
For more details on the Internet of Things revolution, check out these blogs:
The post Difference between Big Data and Internet of Things appeared first on InFocus Blog | Dell EMC Services.
Best Recent Articles on Cloud Computing & Big Data Topics
As we enter a new year, it is time to look back over the past year and resolve to improve upon it. In 2014, we will see more service providers resolve to add more personalization in enterprise technology. Below are seven predictions about what will drive this trend toward personalization.
IT organizations face a growing demand for faster innovation and new applications to support emerging opportunities in social, mobile, growth markets, Big Data analytics, mergers and acquisitions, strategic partnerships, and more. This is great news because it shows that IT continues to be a key stakeholder in delivering business service innovation. However, it also means that IT must deliver new innovation despite flat budgets, while maintaining existing services that grow more complex every day.
Cloud computing is transforming the way businesses think about and leverage technology. As a result, the general understanding of cloud computing has come a long way in a short time. However, there are still many misconceptions about what cloud computing is and what it can do for businesses that adopt this game-changing computing model. In this exclusive Q&A with Cloud Expo Conference Chair Jeremy Geelan, Rex Wang, Vice President of Product Marketing at Oracle, discusses and dispels some of the common myths about cloud computing that still exist today.
Despite the economy, cloud computing is doing well. Gartner estimates the cloud market will double by 2016 to $206 billion. The time for dabbling in the cloud is over! The 14th International Cloud Expo, co-located with 5th International Big Data Expo and 3rd International SDN Expo, to be held June 10-12, 2014, at the Javits Center in New York City, N.Y. announces that its Call for Papers is now open. Topics include all aspects of providing or using massively scalable IT-related capabilities as a service using Internet technologies (see suggested topics below). Cloud computing helps IT cut infrastructure costs while adding new features and services to grow core businesses. Clouds can help grow margins as costs are cut back but service offerings are expanded. Help plant your flag in the fast-expanding business opportunity that is The Cloud, Big Data and Software-Defined Networking: submit your speaking proposal today!
What do you get when you combine Big Data technologies….like Pig and Hive? A flying pig? No, you get a “Logical Data Warehouse.” In 2012, Infochimps (now CSC) leveraged its early use of stream processing, NoSQLs, and Hadoop to create a design pattern which combined real-time, ad-hoc, and batch analytics. This concept of combining the best-in-breed Big Data technologies will continue to advance across the industry until the entire legacy (and proprietary) data infrastructure stack will be replaced with a new (and open) one.
While unprecedented technological advances have been made in healthcare in areas such as genomics, digital imaging and Health Information Systems, access to this information has been not been easy for both the healthcare provider and the patient themselves. Regulatory compliance and controls, information lock-in in proprietary Electronic Health Record systems and security concerns have made it difficult to share data across health care providers.
Cloud Expo, Inc. has announced today that Vanessa Alvarez has been named conference chair of Cloud Expo® 2014. 14th International Cloud Expo will take place on June 10-12, 2014, at the Javits Center in New York City, New York, and 15th International Cloud Expo® will take place on November 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
12th International Cloud Expo, held on June 10–13, 2013 at the Javits Center in New York City, featured four content-packed days with a rich array of sessions about the business and technical value of cloud computing led by exceptional speakers from every sector of the cloud computing ecosystem. The Cloud Expo series is the fastest-growing Enterprise IT event in the past 10 years, devoted to every aspect of delivering massively scalable enterprise IT as a service.
Ulitzer.com announced "the World's 30 most influential Cloud bloggers," who collectively generated more than 24 million Ulitzer page views. Ulitzer's annual "most influential Cloud bloggers" list was announced at Cloud Expo, which drew more delegates than all other Cloud-related events put together worldwide. "The world's 50 most influential Cloud bloggers 2010" list will be announced at the Cloud Expo 2010 East, which will take place April 19-21, 2010, at the Jacob Javitz Convention Center, in New York City, with more than 5,000 expected to attend.
It's a simple fact that the better sales reps understand their prospects' intentions, preferences and pain points during calls, the more business they'll close. Each day, as your prospects interact with websites and social media platforms, their behavioral data profile is expanding. It's now possible to gain unprecedented insight into prospects' content preferences, product needs and budget. We hear a lot about how valuable Big Data is to sales and marketing teams. But data itself is only valuable when it's part of a bigger story, made visible in the right context.
Cloud Expo, Inc. has announced today that Larry Carvalho has been named Tech Chair of Cloud Expo® 2014. 14th International Cloud Expo will take place on June 10-12, 2014, at the Javits Center in New York City, New York, and 15th International Cloud Expo® will take place on November 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Everyone talks about a cloud-first or mobile-first strategy. It's the trend du jour, and for good reason as these innovative technologies have revolutionized an industry and made savvy companies a lot of money. But consider for a minute what's emerging with the Age of Context and the Internet of Things. Devices, interfaces, everyday objects are becoming endowed with computing smarts. This is creating an unprecedented focus on the Application Programming Interface (API) as developers seek to connect these devices and interfaces to create new supporting services and hybrids. I call this trend the move toward an API-first business model and strategy.
We live in a world that requires us to compete on our differential use of time and information, yet only a fraction of information workers today have access to the analytical capabilities they need to make better decisions. Now, with the advent of a new generation of embedded business intelligence (BI) platforms, cloud developers are disrupting the world of analytics. They are using these new BI platforms to inject more intelligence into the applications business people use every day. As a result, data-driven decision-making is finally on track to become the rule, not the exception.
Digital Transformation Blogs