Data mining An overview of techniques and applications - PowerPoint PPT Presentation


PPT – Data mining An overview of techniques and applications PowerPoint presentation | free to download - id: 58a753-OTFiO


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Data mining An overview of techniques and applications


Title: Data mining in the internet age Author: sunita Last modified by: COMPAQ Created Date: 10/16/2000 8:41:51 AM Document presentation format: On-screen Show – PowerPoint PPT presentation

Number of Views:566
Avg rating:3.0/5.0
Slides: 38
Provided by: Sun43
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Data mining An overview of techniques and applications

Data miningAn overview of techniques and
  • Sunita Sarawagi
  • IIT Bombay
  • http//

Data mining
  • Process of semi-automatically analyzing large
    databases to find patterns that are
  • valid hold on new data with some certainity
  • novel non-obvious to the system
  • useful should be possible to act on the item
  • understandable humans should be able to
    interpret the pattern
  • Other names Knowledge discovery in databases,
    Data analysis

Relationship with other fields
  • Overlaps with machine learning, statistics,
    artificial intelligence, databases, visualization
    but more stress on
  • scalability of number of features and instances
  • stress on algorithms and architectures whereas
    foundations of methods and formulations provided
    by statistics and machine learning.
  • automation for handling large, heterogeneous data

  • Mining operations
  • Classification
  • Clustering
  • Association rule mining
  • Sequence mining
  • Two applications
  • Intrusion detection
  • Information extraction

X1 X2 ... Xn Y
2 6.7 BB
5 3.4 .. CA

.. ..

10 0.9 CX -
  • Given, a table D of rows with columns X1..Xn,Y
  • Xi could numeric or string
  • Special attribute Y, the class-label
  • Training
  • Learn a classifier C that can predict the label Y
    in terms of X1,X2..Xn
  • C must hold for
  • examples in D
  • unseen data
  • Application
  • Use C to predict Y for new X-s

10 0.9 CX ?
Automatic loan approval
  • Given old data about customers and payments,
    predict new applicants loan eligibility.

Previous customers
Decision rules
Salary gt 5 L
Age Salary Profession Location Customer type
Good/ bad
Prof. Exec
Age Salary Profession Location
New applicants data
Drug design molecular Bioactivity
  • Predict activity of compounds binding to thrombin
  • Library of compounds included
  • 1909 known molecules (42 actively binding
  • 139,351 binary features describe the 3-D
    structure of each compound
  • 636 new compounds with unknown capacity to bind

Automatic webpage classification
  • Several large categorized search engines
  • Yahoo, Dmoz used in Google/Altavista
  • Web 2 billion pages and only a subset in the
  • Existing taxonomies manually created
  • Need to automatically classify new pages

Several classification methods
  • Choose based on
  • data type (numeric,categorical)
  • number of attributes
  • number of classes
  • number of training examples
  • need for interpretation
  • Regression
  • Decision tree classifier
  • Rule-learners
  • Neural networks
  • Generative models
  • Nearest neighbor
  • Support vector machines

Nearest neighbor
  • Define similarity between instances
  • Find neighbors of new instance in training data
  • K-NN approach assign majority class amongst k
    nearest neighbour
  • weighted regression learn a new regression
    equation by weighting each training instance
    based on distance from new instance
  • Cons
  • Slow during application.
  • No feature selection.
  • Notion of proximity vague
  • Pros
  • Fast training

Decision tree classifiers
  • Widely used learning method
  • Easy to interpret can be re-represented as
    if-then-else rules
  • Approximates function by piece wise constant
  • Does not require any prior knowledge of data
    distribution, works well on noisy data.

Decision trees
  • Tree where internal nodes are simple decision
    rules on one or more attributes and leaf nodes
    are predicted class labels.

Salary lt 20K
Profession teacher
Age lt 30
Algorithm for tree building
  • Greedy top-down construction.

Gen_Tree (Node, data)
make node a leaf?
Selection criteria
Find best attribute and best split on attribute
Partition data on split condition
For each child j of node Gen_Tree (node_j,
Support vector machines
  • Binary classifier find hyper-plane providing
    maximum margin between vectors of the two classes

Support Vector Machines
  • Extendable to
  • Non-separable problems (Cortes Vapnik, 1995)
  • Non-linear classifiers (Boser et al., 1992)
  • Good generalization performance
  • OCR (Boser et al.)
  • Vision (Poggio et al.)
  • Text classification (Joachims)
  • Requires tuning which kernel, what parameters?
  • Several freely available packages SVMTorch

Neural networks
  • Useful for learning complex data like
    handwriting, speech and image recognition

Decision boundaries
Neural network
Classification tree
Linear regression
Bayesian learning
  • Assume a probability model on generation of data.
  • Apply Bayes theorem to find most likely class as
  • Naïve bayes Assume attributes conditionally
    independent given class value

Meta learning methods
  • No single classifier good under all cases
  • Difficult to evaluate in advance the conditions
  • Meta learning combine the effects of the
  • Voting sum up votes of component classifiers
  • Combiners learn a new classifier on the outcomes
    of previous ones
  • Boosting staged classifiers
  • Disadvantage interpretation hard
  • Knowledge probing learn single classifier to
    mimick meta classifier

  • Mining operations
  • Classification
  • Clustering
  • Association rule mining
  • Sequence mining

What is Cluster Analysis?
  • Cluster a collection of data objects
  • Similar to one another within the same cluster
  • Dissimilar to the objects in other clusters
  • Cluster analysis
  • Grouping a set of data objects into clusters
  • Clustering is unsupervised classification no
    predefined classes
  • Typical applications
  • As a stand-alone tool to get insight into data
  • As a preprocessing step for other algorithms

  • Customer segmentation e.g. for targeted marketing
  • Group/cluster existing customers based on time
    series of payment history such that similar
    customers in same cluster.
  • Identify micro-markets and develop policies
  • Image processing
  • Text clustering e.g. scatter/gather
  • Compression

Distance functions
  • Numeric data euclidean, manhattan distances
  • Minkowski metric sum(xi-yi)m(1/m)
  • Larger m gives higher weight to larger distances
  • Categorical data 0/1 to indicate
  • Euclidean distance equal weightage to 1 and 0
  • Hamming distance ( dissimilarity)
  • Jaccard coefficients similarity in 1s/( of 1s)
    (0-0 matches not important
  • data dependent measures similarity of A and B
    depends on co-occurance with C.
  • Combined numeric and categorical dataweighted
    normalized distance

Clustering methods
  • Hierarchical clustering
  • agglomerative Vs divisive
  • single link Vs complete link
  • Partitional clustering
  • distance-based K-means
  • model-based EM
  • density-based

  • Mining operations
  • Classification
  • Clustering
  • Association rule mining
  • Sequence mining
  • Two applications
  • Intrusion detection
  • Information extraction

Intrusion via privileged programs
  • Attacks exploit a loophole in the program to do
    illegal actions
  • Example exploit buffer over-flows to run
  • What to monitor of an executing privileged
    program to detect attacks?

open lseek lstat mmap execve ioctl ioctl close e
xecve close unlink
  • Sequence of system calls
  • S set of all possible system calls 100
  • Mining problem given traces of previous normal
    execution, monitor a new execution and flag
    attack or normal
  • Challenge is it possible to do this given widely
    varying normal conditions?

Detecting attacks on privileged programs
  • Short sequences of system calls made during
    normal execution of system calls are very
    consistent, yet different from the sequences of
    its abnormal executions
  • Each execution a trace of system calls
  • ignore online traces for the moment
  • Two approaches
  • Create dictionary of unique k-windows in normal
    traces, count what fraction occur in new traces
    and threshold.
  • IDS
  • next...

Classification models on k-grams
  • When both normal and abnormal data available
  • class label normal/abnormal
  • When only normal trace,
  • class-labelk-th system call

Learn rules to predict class-label RIPPER
Examples of output RIPPER rules
  • Both-traces
  • if the 2nd system call is vtimes and the 7th is
    vtrace, then the sequence is normal
  • if the 6th system call is lseek and the 7th is
    sigvec, then the sequence is normal
  • if none of the above, then the sequence is
  • Only-normal
  • if the 3rd system call is lstat and the 4th is
    write, then the 7th is stat
  • if the 1st system call is sigblock and the 4th is
    bind, then the 7th is setsockopt
  • if none of the above, then the 7th is open

Experimental results on sendmail
  • The output rule sets contain 250 rules, each
    with 2 or 3 attribute tests
  • Score each trace by counting fraction of
    mismatches and thresholding
  • Summary Only normal traces sufficient to
    detect intrusions

Information extraction
  • Automatically extract structured fields from
    unstructured documents
  • by learning from examples
  • Technology
  • Graph models (Hidden Markov Models)
  • Probabilistic parsers
  • Applications
  • Comparison shopping agents
  • Bibliography databases (citeseer)
  • Address elementization (IIT Bombay)

Problem definition
  • Source concatenation of structured elements with
    limited reordering and some missing fields
  • Example Addresses, bib records

House number
156 Hillside ctype Scenic drive Powai Mumbai
P.P.Wangikar, T.P. Graycar, D.A. Estell, D.S.
Clark, J.S. Dordick (1993) Protein and Solvent
Engineering of Subtilising BPN' in Nearly
Anhydrous Organic Media J.Amer. Chem. Soc. 115,
Learning to segment
  • Given,
  • list of structured elements
  • several examples showing position of structured
    elements in text,
  • Train a model to identify them in unseen text

At top-level a classification problem
  • Issues
  • What are the input features?
  • Build per-element classifiers or a single joint
  • Which type of classifier to use?
  • How much training data is required?

Input features
  • Content of the element
  • Specific keywords like street, zip, vol, pp,
  • Properties of words like capitalization, parts
    of speech, number?
  • Inter-element sequencing
  • Intra-element sequencing
  • Element length

IE with Hidden Markov Models
  • Probabilistic models for IE

HMM Structure
  • Naïve Model One state per element

Mahatma Gandhi Road Near Parkland ...
Mahatma Gandhi Road Near Landmark Parkland
Results Comparative Evaluation
Dataset instances Elements
IITB student Addresses 2388 17
Company Addresses 769 6
US Addresses 740 6
The Nested model does best in all three cases
Mining market
  • Around 20 to 30 mining tool vendors
  • Major tool players
  • SASs Enterprise Miner.
  • IBMs Intelligent Miner,
  • SGIs MineSet,
  • All pretty much the same set of tools
  • Many embedded products
  • fraud detection
  • electronic commerce applications,
  • health care,
  • customer relationship management Epiphany