Managing your Assets with Big Data Tools (1) - PowerPoint PPT Presentation

About This Presentation
Title:

Managing your Assets with Big Data Tools (1)

Description:

Karthik provided a comprehensive understanding of available ecosystem tools and how they can be used to perform data engineering and data analytics. Karthik covers the following topics in his presentation: • Establishment of complete data pipeline using big data ecosystem tools. • Tackling of high velocity streams using various stream processing engines on cloud and performing Real Time analytics. • Integration of big data ecosystem for data analysis using SAMOA , R and Mahout. • Deployments of big data environments on the cloud. See more at – PowerPoint PPT presentation

Number of Views:108

less

Transcript and Presenter's Notes

Title: Managing your Assets with Big Data Tools (1)


1
Managing your Assets with Big Data Tools
  • Karthigai Muthu, MachinePulse

2
Agenda
  • Big Data value proposition
  • Big Data Technology Stack

3
Hype Cycle for Emerging Technologies
Source Wikipedia
4
Sources of data
12 TBs of tweet data every day
? TBs ofdata every day
25 TBs oflog data every day
5
What makes Data Big
Characteristics Description Attributes Drivers
Volume The amount of data generated or intensify that must be ingested, analyzed and managed to make decision based on complete data analysis Exabyte (EB) Zettabyte (ZB) Yottabyte (YB) Increase in data sources Higher resolution sensors Scalable infrastructure
Velocity How fast the data is being produced and changed and the speed at which is transformed into insight Batch Near real time Real time and Streams Rapid feedback loop Improved throughput connectivity Competitive advantage Pre-computed information
Variety The degree of diversity of data from sources both inside and outside an organization Degree of structure Complexity M2M/IoT Social Media Genomics Video and Mobile
Veracity The quality and provenance of data Consistency Completeness Ambiguity Integrity Cost Need of traceability and justification
6
Big Datas Greatest Power Predictive Analytics
7
Whats driving Big Data
8
The Evolution of Business Intelligence
Big Data Real Time Single View Graph
Databases
Interactive Business Intelligence In-memory
RDBMS QlikView, Tableau,HANA
Speed
Scale
BI Reporting OLAP Data warehouse Business
Objects, SAS, Informatica, Cognos other SQL
Reporting Tools
Speed
Scale
Big Data Batch Processing Distributed Data
Store Hadoop/Spark HBase/Cassandra/MongoDB
1990s
2000s
2010s
9
Solving business problem with big data
10
Formulation of big data strategy
11
Companies Market share in Big Data
12
Big Data Investments
13
Priority for big data across industry
14
Are you aware the risk of not implementing Big
Data in your company
15
Big data changed connected things to Internet of
Everything(IoE)
16
How the industry can leverage from big data
17
Challenges in implementing the big data
18
Returns of Investment(ROI)
19
How do companies get MORE from big data
  • Merge
  • Optimize
  • Respond
  • Empower

20
Are you planning to launch your new product.
21
Customer 360
Social Media
Banking Finance
Our Known History
Gaming
Purchase
Entertain
Customer
22
Real-Time Analytics/Decision Requirement
Product Recommendations that are Relevant
Compelling
Friend Invitations to join a Game or
Activity that expands business
Influence Behavior
Improving the Marketing Effectiveness of a
Promotion while it is still in Play
Customer
Learning why Customers Switch to competitors
and their offers in time to Counter
Preventing Fraud as it is Occurring
preventing more proactively
23
IoTBig Data IoE(Internet-of-Everything)
24
Role of Big Data in M2M/IoT
  • Big Data is a factor that will, to a large
    extent, determine the future growth rate in the
    M2M industry
  • M2M will connect increasingly more nodes that
    will provide data from endpoints.
  • Data will be more granular, more frequent, and
    more accurate, with bigger data sets or even live
    data streams
  • Large volume of endpoint connections IPv4
    addressing scheme cant accommodate
    everything(sensors, smart phones, smart
    factories, smart grids, smart vehicles,
    controllers, meters ) that it requires IPv6
  • IoE Convergence of IoT, Big Data Analytics
    ,Cloud Computing and other technologies is
    collectively called as Internet of Everything

25
Challenges of Big Data in M2M/IoT
  • Meeting the need for speed
  • Data understanding
  • Maintaining data quality
  • Displaying the meaningful result

26
IoT/M2M Applications..
27
Big Data Use Cases IoT/M2M
  • Personal IoT the scope is a single person, such
    as a smartphone equipped with GPS sensor or a
    fitness device that measures the heart rate. This
    is one of the fastest growing, consumer-oriented
    areas of IoT.
  • Group IoT the scope is a fairly small group of
    people, such as a family in a smart house,
    co-workers in a van or a group of tourists. This
    is one of the most challenging areas and is still
    in its early phase.
  • Community IoT the scope is a large group of
    people, potentially thousands and more usually
    this is in a public infrastructure context, such
    as smart cities or smart roads. This is a young
    and potentially promising IoT area.
  • Industrial IoT the scope can be within an
    organization (smart factory) or between
    organizations (retailer supply chain). This is
    arguably the most established and mature part of
    IoT.

28
Big Data Use cases IoT/M2M
  • Agriculture - sensors can be deployed on farm
    machinery in order to provide data about the
    equipment, soil temperature, moisture, etc.
  • Buildings/Smart Homes - Building sensors be used
    to help facility managers become more proactive
    about ensuring that their buildings operate at
    peak efficiency.
  • Communities Smart cities make use of parking
    space availability systems, intelligent traffic
    monitoring systems, intelligent highways,
    weather-adaptive street lighting, and more.
  • Healthcare Infant monitors, smart diapers,
    pills with ingestible sensors are just some of
    the IOT-based devices.
  • Manufacturing factories with sensors can
    improve operations, product quality, and decrease
    safety hazards.
  • Smartphones can control everything from door
    locks, thermostats, light bulbs, vacuum cleaners,
    and more.
  • Utilities smart water meters can be used to
    reduce water leaks. Smart electric grids can
    adjust rates depending on usage.
  • Wearables Smart watches, fitness trackers and
    health monitors may become primary source for
    human-related data, and can also be used in
    sports, retail, travel and manufacturing.

29
Benefits of Big Data Analytics in M2M/IoT
  • Device Maintenance
  • a. Time for next patch upgrade
  • b. Energy management
  • c. Inventory management and track replacement
  • 2. Proactive Healthcare
  • Capture and analyze real time data from medical
    monitors to predict potential health problems
    before patients manifest clinical signs of
    infection.
  • 3. Monetize Machine Data
  • a. Monitor performance, usage and capacity
    details to uncover up-sell and cross-sell
    opportunities
  • b. Maximize the lifespan and performance of high
    value medical assets

30
Benefits of Big Data Analytics cont..
  • 4. Optimize Support Operations
  • a. Reduce MTTR and support escalations
  • b. Preempt failures with proactive support
  • c. Troubleshoot with accurate information
  • d. Proactive consultation to customers on
    approaching expiry dates

31
Big Data Analytics Stack
32
Lamda Architecture
33
(No Transcript)
34
Batch vs. Real-Time processing
  • Batch processing
  • - Gathering of data and processing as a group at
    one time.
  • - Jobs run to completion
  • - Data might be out of date
  • Real-time processing
  • - Processing of data that takes place as the
    information is being entered.
  • - Run for ever

35
Storm
  • Apache Storm is a free and open source
    distributed real-time computation system.
  • Storm makes it easy to reliably process unbounded
    streams of data, doing for real-time processing
    what Hadoop did for batch processing

36
Storm Is
  • Stream Processing
  • Fast
  • Scalable
  • Fault Tolerant
  • Reliable

37
Tuple
38
Streams
39
Spouts
40
Bolts
41
Topologies
42
Reliable Processing
43
Reliable Processing
44
Stream Grouping
  • Groupings are used to decide to which task in
    the
  • subscribing bolt (group) a tuple is sent.
  • Possible Groupings
  • - Shuffle
  • - Fields
  • - All
  • - Global
  • - None
  • - Direct
  • - Local or Shuffle

45
Storm Cluster View
46
Fault Tolerance
47
Fault Tolerance
48
Fault Tolerance
49
Fault Tolerance
50
Fault Tolerance
51
Parallelism
52
Parallelism
53
Apache Storm Real-time -Use cases
Segment Prevent Use Cases Optimize Use Cases
Financial Services Securities fraud Operational risks compliance violations Order routing Pricing
Telecom Security breaches Network outages Bandwidth allocation Customer service
Retails Shrinkage Stock outs Offers Pricing
Manufacturing Preventative maintenance Quality assurance Supply chain optimization Reduced plant downtime
Transportation Driver monitoring Predictive maintenance Routes Pricing
Web Application failures Operational Issues Personalized content
54
The End
Write a Comment
User Comments (0)
About PowerShow.com