Data Mining and Knowledge Acquizition Chapter 8 Data Preprocessing

1
Data Mining and Knowledge Acquizition Chapter
8 Data Preprocessing

• Summer
• 2005

2
Chapter 8 Data Preprocessing

• Why preprocess the data?
• Data cleaning
• Data integration and transformation
• Data reduction
• Discretization and concept hierarchy generation
• Time-dependent data
• Summary

3
Why Data Preprocessing?

• Data in the real world is dirty
• incomplete lacking attribute values, lacking
  certain attributes of interest, or containing
  only aggregate data
• e.g., occupation
• noisy containing errors or outliers
• e.g., Salary-10
• inconsistent containing discrepancies in codes
  or names
• e.g., Age42 Birthday03/07/1997
• e.g., Was rating 1,2,3, now rating A, B, C
• e.g., discrepancy between duplicate records

4
Why Is Data Dirty?

• Incomplete data comes from
• n/a data value when collected
• different consideration between the time when the
  data was collected and when it is analyzed.
• human/hardware/software problems
• Noisy data comes from the process of data
• collection
• entry
• transmission
• Inconsistent data comes from
• different data sources
• functional dependency violation

5
Why Is Data Preprocessing Important?

• No quality data, no quality mining results!
• Quality decisions must be based on quality data
• e.g., duplicate or missing data may cause
  incorrect or even misleading statistics.
• Data warehouse needs consistent integration of
  quality data
• Data extraction, cleaning, and transformation
  comprises the majority of the work of building a
  data warehouse

6
Multi-Dimensional Measure of Data Quality

• A well-accepted multidimensional view
• Accuracy
• Completeness
• Consistency
• Timeliness
• Believability
• Value added
• Interpretability
• Accessibility
• Broad categories
• intrinsic, contextual, representational, and
  accessibility.

7
Major Tasks in Data Preprocessing

• Data cleaning
• Fill in missing values, smooth noisy data,
  identify or remove outliers, and resolve
  inconsistencies
• Data integration
• Integration of multiple databases, data cubes, or
  files
• Data transformation
• Normalization and aggregation
• Data reduction
• Obtains reduced representation in volume but
  produces the same or similar analytical results
• Data discretization
• Part of data reduction but with particular
  importance, especially for numerical data

8
Forms of data preprocessing
9
Chapter 3 Data Preprocessing

• Why preprocess the data?
• Data cleaning
• Data integration and transformation
• Data reduction
• Discretization and concept hierarchy generation
• Time-dependent data
• Summary

10
Data Cleaning

• Importance
• Data cleaning is one of the three biggest
  problems in data warehousingRalph Kimball
• Data cleaning is the number one problem in data
  warehousingDCI survey
• Data cleaning tasks
• Fill in missing values
• Identify outliers and smooth out noisy data
• Correct inconsistent data
• Resolve redundancy caused by data integration

11
Missing Data

• Data is not always available
• E.g., many tuples have no recorded value for
  several attributes, such as customer income in
  sales data
• Missing data may be due to
• equipment malfunction
• inconsistent with other recorded data and thus
  deleted
• data not entered due to misunderstanding
• certain data may not be considered important at
  the time of entry
• not register history or changes of the data
• Missing data may need to be inferred.

12
How to Handle Missing Data?

• Ignore the tuple usually done when class label
  is missing (assuming the tasks in
  classificationnot effective when the percentage
  of missing values per attribute varies
  considerably.
• Fill in the missing value manually tedious
  infeasible?
• Use a global constant to fill in the missing
  value e.g., unknown, a new class?!
• Use the attribute mean,median or mode to fill in
  the missing value
• Use the attribute mean for all samples belonging
  to the same class to fill in the missing value
  smarter
• Use the most probable value to fill in the
  missing value inference-based such as Bayesian
  formula or decision tree

13
Most Probable Value

• Use the most probable value to fill in the
  missing value inference-based such as Bayesian
  formula or decision tree
• develop a submodel to predict the category or
  numerical value of the missing case
• income f(education,sex, age,...)
• A neural network model
• K-NN model
• Decision tree
• Regression
• ...

14
An Extreme Case

• Suppose all values of an attribute is missing
• This attribute may be considered as the unknown
  label in unsupervised learning
• Assigning an object into a cluster can be thought
  of filling the missing values of an attribute

15
Time Series and Spacial Data

• Filling missing values in time series and
  spatial data needs a different tratement

Stock index
x
x data exisits o missing
x
x
o
x
o
x
x
o
x
Time in days
16
Noisy Data

• Noise random error or variance in a measured
  variable
• Incorrect attribute values may due to
• faulty data collection instruments
• data entry problemshuman or computer error
• data transmission problems
• technology limitation
• inconsistency in naming convention
• duplicate records
• Some noice is inavitable and due to variables or
  factors that can not be measured

17
How to Handle Noisy Data?

• Binning method
• first sort data and partition into (equi-depth)
  bins
• then one can smooth by bin means, smooth by bin
  median, smooth by bin boundaries, etc.
• Clustering
• detect and remove outliers
• Combined computer and human inspection
• detect suspicious values and check by human
• Regression
• smooth by fitting the data into regression
  functions

18
Simple Discretization Methods Binning

• Equal-width (distance) partitioning
• It divides the range into N intervals of equal
  size uniform grid
• if A and B are the lowest and highest values of
  the attribute, the width of intervals will be W
  (B-A)/N.
• The most straightforward
• But outliers may dominate presentation
• Skewed data is not handled well.
• Equal-depth (frequency) partitioning
• It divides the range into N intervals, each
  containing approximately same number of samples
• Managing categorical attributes can be tricky.

19
Binning Methods for Data Smoothing

• Sorted data for price (in dollars) 4, 8, 9,
  15, 21, 21, 24, 25, 26, 28, 29, 34
• Partition into (equi-depth) bins
• - Bin 1 4, 8, 9, 15
• - Bin 2 21, 21, 24, 25
• - Bin 3 26, 28, 29, 34
• Smoothing by bin means
• - Bin 1 9, 9, 9, 9
• - Bin 2 23, 23, 23, 23
• - Bin 3 29, 29, 29, 29
• Smoothing by bin boundaries
• - Bin 1 4, 4, 4, 15
• - Bin 2 21, 21, 25, 25
• - Bin 3 26, 26, 26, 34

20
Cluster Analysis
21
Regression
y
Y1
y x 1
Y1
x
X1
22
Example use of regression to determine outliers

• Problem predict the yearly spending of AE
  customers from their income.
• Y yearly spending in YTL
• X average montly income in YTL

23
Data set
X income, Y spending First data is probabliy an
outlier But examining only spending data that is
Distribution of Y 200 spending of the
first Customer can not be identified as an
outlier

• X Y
• 50 200
• 100 50
• 200 70
• 250 80
• 300 85
• 400 120
• 500 130
• 700 150
• 750 150
• 800 180
• 1000 200

24
Regression
y
Y1200
y 0.15x 20
Y1
x
X150
X101000
Examine residuals Y1-Y1 is high But Y10-Y10 is
low So data point 1 is an outlier
inccome
25
Notes

• Some methods are used for both smoothing and data
  reduction or discretization
• binning
• used in decision tress to reduce number of
  categories
• concept hierarchies
• Example price a numerical variable
• in to concepts as expensive, moderately prised,
  expensive

26
Inconsistent Data

• Inconsistent data may due to
• faulty data collection instruments
• data entry problemshuman or computer error
• data transmission problems
• Chang in scale over time
• 1,2,3 to A, B. C
• inconsistency in naming convention
• Data integration
• Different units used for the same variable
• TL or dollar
• Value added tax included in one source not in
  other
• duplicate records

27
Chapter 2 Data Preprocessing

• Why preprocess the data?
• Descriptive data summarization
• Data cleaning
• Data integration and transformation
• Data reduction
• Discretization and concept hierarchy generation
• Summary

28
Mining Data Dispersion Characteristics

• Motivation
• To better understand the data central tendency,
  variation and spread
• Data dispersion characteristics
• median, max, min, quantiles, outliers, variance,
  etc.
• Numerical dimensions correspond to sorted
  intervals
• Data dispersion analyzed with multiple
  granularities of precision
• Boxplot or quantile analysis on sorted intervals
• Dispersion analysis on computed measures
• Folding measures into numerical dimensions
• Boxplot or quantile analysis on the transformed
  cube

29
Measuring the Central Tendency

• Mean (algebraic measure)
• Weighted arithmetic mean
• Trimmed mean chopping extreme values
• Median A holistic measure
• Middle value if odd number of values, or average
  of the middle two values otherwise
• Estimated by interpolation (for grouped data)
• Mode
• Value that occurs most frequently in the data
• Unimodal, bimodal, trimodal
• Empirical formula

30
Symmetric vs. Skewed Data

• Median, mean and mode of symmetric, positively
  and negatively skewed data

31
Measuring the Dispersion of Data

• Quartiles, outliers and boxplots
• Quartiles Q1 (25th percentile), Q3 (75th
  percentile)
• Inter-quartile range IQR Q3 Q1
• Five number summary min, Q1, M, Q3, max
• Boxplot ends of the box are the quartiles,
  median is marked, whiskers, and plot outlier
  individually
• Outlier usually, a value higher/lower than 1.5 x
  IQR
• Variance and standard deviation
• Variance s2 (algebraic, scalable computation)
• Standard deviation s is the square root of
  variance s2

32
Properties of Normal Distribution Curve

• The normal (distribution) curve
• From µs to µs contains about 68 of the
  measurements (µ mean, s standard deviation)
• From µ2s to µ2s contains about 95 of it
• From µ3s to µ3s contains about 99.7 of it

33
Boxplot Analysis

• Five-number summary of a distribution
• Minimum, Q1, M, Q3, Maximum
• Boxplot
• Data is represented with a box
• The ends of the box are at the first and third
  quartiles, i.e., the height of the box is IRQ
• The median is marked by a line within the box
• Whiskers two lines outside the box extend to
  Minimum and Maximum

34
Positively and Negatively Correlated Data
35
Not Correlated Data
36
Graphic Displays of Basic Statistical Descriptions

• Histogram (shown before)
• Boxplot (covered before)
• Quantile plot each value xi is paired with fi
  indicating that approximately 100 fi of data
  are ? xi
• Quantile-quantile (q-q) plot graphs the
  quantiles of one univariant distribution against
  the corresponding quantiles of another
• Scatter plot each pair of values is a pair of
  coordinates and plotted as points in the plane
• Loess (local regression) curve add a smooth
  curve to a scatter plot to provide better
  perception of the pattern of dependence

37
Chapter 8 Data Preprocessing

• Why preprocess the data?
• Data cleaning
• Data integration and transformation
• Data reduction
• Discretization and concept hierarchy generation
• Time-dependent data
• Summary

38
Data Integration

• Data integration
• combines data from multiple sources into a
  coherent store
• Schema integration
• integrate metadata from different sources
• Entity identification problem identify real
  world entities from multiple data sources, e.g.,
  A.cust-id ? B.cust-
• Detecting and resolving data value conflicts
• for the same real world entity, attribute values
  from different sources are different
• possible reasons different representations,
  different scales, e.g., metric vs. British units
• price may include value added tax in one source
  and not include in the other
• Name are entered by different convensions
• name lastname or lastname name or ...

39
Handling Redundant Data in Data Integration

• Redundant data occur often when integration of
  multiple databases
• The same attribute may have different names in
  different databases
• One attribute may be a derived attribute in
  another table, e.g., annual revenue
• Redundant data may be able to be detected by
  correlational analysis
• rx,y ?ni1(xi-mean_x)(yi-mean_y)/(n-1)?x?y
• where ?x,?y are standard deviation of X and Y
• as r?1 positive correlation x? y? or x? y?, one
  is redundant
• r ? 0, X and X are independent
• r ? -1 negative correlation x? y? or x? y? one
  is redundant

40
Positively and Negatively Correlated Data
41
Not Correlated Data
42
Exercise

• Construct a data set or a functional relationship
  of two variables X and Y where there is a perfect
  relation between X and Y
• knowing value of X, Y corresponding to that X can
  be predicted with no error
• but correlation coefficient between X and Y is
• ZERO

43
Correlation and Causation

• Correlation does not imply causality
• of hospitals and of car-theft in a city are
  correlated
• Both are causally linked to the third variable
  population

44
Data Transformation Normalization

• min-max normalization

• When a score is outside the new range the
  algorithm may give an error or warning message
• in the presence of outliers regular observations
  are squized in to a small interval

45
Data Transformation Normalization

• z-score normalization

Good in handling outliers as the new range is
between -?to ? no out of range error
46
Data Transformation Normalization

• Decimal scaling

Where j is the smallest integer such that Max(
)lt1

• Ex max v984 then j 3 v0.984
• preserves the appearance of figures

47
Data TransformationLogarithmic Transformation

• Logarithmic transformations
• Y logY
• Used in some distance based methods
• Clustering
• For ratio scaled variables
• E.g. weight, TL/dollar
• Distance between objects is related to percentage
  changes rather then actual differences

48
Linear transformations

• Note that linear transformations preserve the
  shape of the distribution
• Whereas non linear transformations distors the
  distribution of data
• Example
• Logistic function f(x) 1/(1exp(-x))
• Transforms x between 0 and 1
• New data are always between 0-1

49
Exercise

• Device a transformation such that new data are
  always between 0-1 interval
• but preserve the shape of the sample distribution
  as much as possible

50
Attribute/feature construction

• Automatic attribute generation
• Using product or and opperations
• E.g. a regression model to explain spending of a
  customer shown by Y
• Using X1 age, X2 income, ...
• Y a b1X1 b2X2 a linear model
• a,b1,b2 are parameters to be estimated
• Y a b1X1 b2X2 cX1X2
• a nonlinear model a,b1,b2,c are parameters to be
  estimated but linear in parameters as
• X1X2 can be directly computed from income and
  age

51
Ratios or differences

• Define new attributes
• E.g.
• Real variables in macroeconomics
• Real GNP
• Financial ratios
• Profit revenue cost

52
Chapter 8 Data Preprocessing

• Why preprocess the data?
• Data cleaning
• Data integration and transformation
• Data reduction
• Discretization and concept hierarchy generation
• Time-dependent data
• Summary

53
Data Reduction Strategies

• Warehouse may store terabytes of data Complex
  data analysis/mining may take a very long time to
  run on the complete data set
• Data reduction
• Obtains a reduced representation of the data set
  that is much smaller in volume but yet produces
  the same (or almost the same) analytical results
• Data reduction strategies
• Data cube aggregation
• Dimensionality reductionremove unimportant
  attributes
• Data Compression
• Numerosity reductionfit data into models
• Discretization and concept hierarchy generation

54
Credit Card Promotion Database
55
Data Reduction

• Eliminate redandent attributes
• correlation coefficients show that
• Watch and magazine are associated
• Eliminate one
• Combine variables to reduce number of independent
  variables
• Principle component analysis
• Sampling reduce number of cases (records) by
  sampling from secondary storage
• Discretization
• Age to categorical values

56
Dimensionality Reduction

• Feature selection (i.e., attribute subset
  selection)
• Select a minimum set of features such that the
  probability distribution of different classes
  given the values for those features is as close
  as possible to the original distribution given
  the values of all features
• reduce of patterns in the patterns, easier to
  understand
• Heuristic methods (due to exponential of
  choices)
• step-wise forward selection
• step-wise backward elimination
• combining forward selection and backward
  elimination
• decision-tree induction

57
Example for irrelevent features
Average spending of a customer by income and age
Droping age Expain spending by Income only
Does not change results significantly
58
Example of Decision Tree Induction
Initial attribute set A1, A2, A3, A4, A5, A6
A4 ?
A6?
A1?
Class 2
Class 2
Class 1
Class 1
Reduced attribute set A1, A4, A6
59
Heuristic Feature Selection Methods

• There are 2d possible sub-features of d features
• Several heuristic feature selection methods
• Best single features under the feature
  independence assumption choose by significance
  tests.
• Best step-wise feature selection
• The best single-feature is picked first
• Then next best feature condition to the first,
  ...
• Step-wise feature elimination
• Repeatedly eliminate the worst feature
• Best combined feature selection and elimination
• Optimal branch and bound
• Use feature elimination and backtracking

60
Example Economic indicator problem

• There are tens of macroeconomic variables
• say totally 45
• Which ones is the best predictor for inflation
  rate three months ahead?
• Develop a simple model to predict inflation by
  using only a couple of those 45 macro variables
• Best-stepwise feature selection
• The single macro variable predicting inflation
  among the 45 is seleced first try 45 models say
  /TL
• repeat
• The k th variable is entered among 45-(k-1)
  variables
• Stop at some point introducing new variables

61
Example cont.

• Best feature elimination
• Develop a model including all 45 variables
• Remove just one of them try 45 models each
  excluding just one out of 45 variables
• repeat
• Continue eliminating a new variable at each step
• Unitl a stoping criteria
• Rearly used comared to feature selection

62
Principal Component Analysis

• Given N data vectors from k-dimensions, find c lt
  k orthogonal vectors that can be best used to
  represent data
• The original data set is reduced to one
  consisting of N data vectors on c principal
  components (reduced dimensions)
• Each data vector is a linear combination of the c
  principal component vectors
• Works for numeric data only
• Used when the number of dimensions is large

63
Principal Component Analysis
X2
Y1
Y2

X1

64

• X Nk matrix of data in original form Xj is j th
  column of data matrix X where j 1,.,k
• xij is the j th variable of i th observation
• First normalize all variables X1..Xk to Z1.. Zk
• such that mean of Zj j1,..,k 0 E(Zi) 0
• If the scales of variables are differnt bring the
  scales of all variables to same units apply
• Zi (Xi-mean_X)/std_X

65

• The first principle component a1 is a linear
  combination of the data Z
• y1 Za1 where y1 is N1 vector Z is NK and
  ak1
• y1,1 z1,1a1,1 z1,2a2,1... z1,kak,1
• y2,1 z2,1a1,1 z2,2a2,1... z2,kak,1

66
Principal Component Analysis
X2
Y1
Y2

• Coordinate of Z1 is
• Z1,Z2 in
• when using principle
• components
• Z y1,y2 expressed in
• unit vectors a1 a2

• a1

• a2

Z1
67
Histograms

• A popular data reduction technique
• Divide data into buckets and store average (sum)
  for each bucket
• Can be constructed optimally in one dimension
  using dynamic programming
• Related to quantization problems.

68

• Equiwidht the width of each bucket range is
  uniform
• Equidepth (equiheight)each bucket contains
  roughly the same number of continuous data
  samples
• V-Optimalleast variance
• histogram variance is the is a weighted sum of
  the original values that each bucket represents
• bucket weight values in bucket

69
MaxDiff

• MaxDiff make a bucket boundry between adjacent
  values if the difference is one of the largest k
  differences
• xi1 - xi gt max_k-1(x1,..xN)

70
V-optimal design

• Sort the values
• assign equal number of values in each bucket
• compute variance
• repeat
• change buckets of boundary values
• compute new variance
• until no reduction in variance
• variance (n1Var1n2Var2...nkVark)/N
• N n1n2..nk,
• Vari ?nij1(xj-x_meani)2/ni,
• Note that V-Optimal in one dimension is
  equivalent to K-means clustering in one dimension

71
Sampling

• Allow a mining algorithm to run in complexity
  that is potentially sub-linear to the size of the
  data
• Choose a representative subset of the data
• Simple random sampling may have very poor
  performance in the presence of skew
• Develop adaptive sampling methods
• Stratified sampling
• Approximate the percentage of each class (or
  subpopulation of interest) in the overall
  database
• Used in conjunction with skewed data
• Sampling may not reduce database I/Os (page at a
  time).

72
Sampling methods

• Simple random sample without replacement (SRSWOR)
  of size n
• n of N tuples from D nltN
• P(drawing any tuple)1/N all are equally likely
• Simple random sample with replacement (SRSWR) of
  size n
• each time a tuple is drawn from D, it is recorded
  and then replaced
• it may be drawn again

73
Sampling methods cont.

• Cluster Sample if tuples in D are grouped into M
  mutually disjoint clusters then an SRS of m
  clusters can be obtained where m lt M
• Tuples in a database are retrieved a page at a
  time
• Each pages can be considered as a cluster
• Stratified Sample if D is divided into mutually
  disjoint parts called strata.
• Obtain a SRS at each stratum
• a representative sample when data are skewed
• Ex customer data a stratum for each age group
• The age group having the smallest number of
  customers will be sure to be presented

74
Confidence intervals and sample size

• Cost of obtaining a sample is propotional to the
  size of the sample, n
• Specifying a confidence interval and
• you should be able to determine n number of
  samples required so that the sample mean will be
  within the confidence interval with (1-p)
  confident
• n is very small compared to the size of the
  database N
• nltltN

75
Sampling
SRSWOR (simple random sample without
replacement)
SRSWR
76
Sampling
Cluster/Stratified Sample
Raw Data
77
Chapter 3 Data Preprocessing

• Why preprocess the data?
• Data cleaning
• Data integration and transformation
• Data reduction
• Discretization and concept hierarchy generation
• Time-dependent data
• Summary

78
Time Series Data

• A time series database consists of sequences of
  values or events changing with time.
• The values are typically measured at equal time
  intervals
• Examples
• daily closing values of stock market index
• sales of products
• economic variables GNP, exchange rates...

79
Mining Time-Series and Sequence Data
Time-series plot
80
Objective (1)

• forecasting future unknown values based on
  historical observed data
• one step ahead
• predict Yt1 from YtYt-1, Yt-2...
• multiple steps ahead
• predict Yt1Yt2 Yt3 .. from YtYt-1, Yt-2...
• univariate just one variable
• forecast of TL/ based on its own values

81
Objective (2)

• multivariate more then one variable is
  forecasted simultaneously
• inflation, stock index, exchange rate, interest
  rates influence each other
• model based forecasts

82
Data preprosessing

• Handling missing values
• methods for cross-sectional data are not
  applicable
• fill missing values by smoothing
• Aggregation disaggregation problems
• frequency of raw data and desired forecast
  frequency may be different
• aggregation form low level to high level
• sum,average, minimum,maximum, last,first
• which function is used depends on the type of data

83

• disaggregation from high to low frequency
• different concept from OLAPs drill down where low
  level data is already available
• here low level data is not available
• GNP is available at the quarterly level
• but may be needed in monthly forecast of GNP
• Methods
• constant value is used
• equally partitioned
• linear smoothing
• cubic smoothing

84
Components of time series data

• long-term or trend
• indicate the general direction in which a time
  series variable is moving over a long interval of
  time
• a line or curve
• Cyclic movements or cyclic variations
• long term oscillations about a trend line or
  curve may or may not be periodic

85
Components of time series data

• Seasonal movements or variations
• due to events that recur annually
• sales of beverages varies from season to season
• fuel oil sales in winter are higher than in
  summer
• identical or nearly identical patterns during
  corresponding months or seasons of successive
  years
• Irregular or random movements
• sporadic motion of time series due to random or
  chance events

86
Decomposition of time series

• product of the four variables
• Yi TiCiSiIi,
• or sum of the four variables
• Yi Ti Ci Si Ii,
• Identify each components and try to forecast each
  seperately
• The I component can not be forecasted or
  predicted as it is irregular

87
How to determine trend

• A moving average of order m
• (y1y2...ym)/m
• a moving average tends to reduce the amount of
  variation present in the data set
• eliminates unwanted fluctuations
• smoothing of time series
• Weighted moving average of order m

88
Example

• original data
• 3 7 2 0 4 5 9 7 2
• MA(3) 4 3 2 3 6 7 6
• weighed
• MA(3) 5.5 2.5 1 3.5 5.5 8 6.5
• 1 4 1
• the first WMA value
• (134712)/(141)5.5
• MA loses the data at the beginning and end of a
  series may sometimes

89
Detecting trend

• By moving averages
• cyclic, seasonal and irregular patterns in the
  data can be eliminated
• resulting only the trend movement
• Free-hand method
• approximate curve or line is drawn to fit a set
  of data based on the users own judgement
• costly and not reliable

90
least square method

• Fit a line by regression
• Yt a bt a linear trend
• Yt a bt ct2 a quadratic trend
• Yt atb,convert into linear by logarithmic
  transformation
• lnYt lnablnt
• Yt aebt, exponential
• lnYt lna bt

91
Seasonal variations

• identify and remove seasonal variations
• deseasonalize the data for trend and cyclic
  analysis
• A seasonal index set of number showing the
  relative values of a variable during the months
  of a year
• yt yt/st,
• where yt is deseasonalized variable
• st is seasonal index value at time t

92

• Estimation of seasonal variations
• Seasonal index
• Set of numbers showing the relative values of a
  variable during the months of the year
• E.g., if the sales during October, November, and
  December are 80, 120, and 140 of the average
  monthly sales for the whole year, respectively,
  then 80, 120, and 140 are seasonal index numbers
  for these months
• Deseasonalized data
• Data adjusted for seasonal variations
• E.g., divide the original monthly data by the
  seasonal index numbers for the corresponding
  months

93
(No Transcript)
94

• Visually inspect the data
• to identify trend cyclic or seasonal components

95
(No Transcript)
96
Mining Time-Series and Sequence Data

• Time-series database
• Consists of sequences of values or events
  changing with time
• Data is recorded at regular intervals
• Characteristic time-series components
• Trend, cycle, seasonal, irregular
• Applications
• Financial stock price, inflation
• Biomedical blood pressure
• Meteorological precipitation

97
Mining Time-Series and Sequence Data Trend
analysis

• A time series can be illustrated as a time-series
  graph which describes a point moving with the
  passage of time
• Categories of Time-Series Movements
• Long-term or trend movements (trend curve)
• Cyclic movements or cycle variations, e.g.,
  business cycles
• Seasonal movements or seasonal variations
• i.e, almost identical patterns that a time series
  appears to follow during corresponding months of
  successive years.
• Irregular or random movements

98
Estimation of Trend Curve

• The freehand method
• Fit the curve by looking at the graph
• Costly and barely reliable for large-scaled data
  mining
• The least-square method
• Find the curve minimizing the sum of the squares
  of the deviation of points on the curve from the
  corresponding data points
• The moving-average method
• Eliminate cyclic, seasonal and irregular patterns
• Loss of end data
• Sensitive to outliers

99
Discovery of Trend in Time-Series (1)

• Estimation of seasonal variations
• Seasonal index
• Set of numbers showing the relative values of a
  variable during the months of the year
• E.g., if the sales during October, November, and
  December are 80, 120, and 140 of the average
  monthly sales for the whole year, respectively,
  then 80, 120, and 140 are seasonal index numbers
  for these months
• Deseasonalized data
• Data adjusted for seasonal variations
• E.g., divide the original monthly data by the
  seasonal index numbers for the corresponding
  months

100
Discovery of Trend in Time-Series (2)

• Estimation of cyclic variations
• If (approximate) periodicity of cycles occurs,
  cyclic index can be constructed in much the same
  manner as seasonal indexes
• Estimation of irregular variations
• By adjusting the data for trend, seasonal and
  cyclic variations
• With the systematic analysis of the trend,
  cyclic, seasonal, and irregular components, it is
  possible to make long- or short-term predictions
  with reasonable quality

101
Chapter 3 Data Preprocessing

• Why preprocess the data?
• Data cleaning
• Data integration and transformation
• Data reduction
• Discretization and concept hierarchy generation
• Summary

102
Summary

• Data preparation is a big issue for both
  warehousing and mining
• Data preparation includes
• Data cleaning and data integration
• Data reduction and feature selection
• Discretization
• A lot a methods have been developed but still an
  active area of research