Using MDM as a Practical Approach to Get Started in Data Governance Todd Goldman: VP Products and Marketing

1

Using MDM as a Practical Approach to Get Started
in Data Governance Todd Goldman VP Products
and Marketing
2
Precursor to Data Governance is Data Management
3
Data Governance Manifesto

• Data should be
• Understood
• Secure
• Consistent
• Accessible
• Managed

Data Governance
4
Governance Requires a Foundation of Understanding

• If you dont know how data in different systems
  is related How can you make sure they are
  consistent (MDM)?
• How can you measure overall data quality?
• How can you measure the quality of business
  rules?
• If you dont know where the sensitive data
  is How can you protect it?
• If your data is not secure, consistent and
  accessible What does it mean to manage it?

5
Even if you understand your data
landscapestarting data governance is
difficult.It is a new religion.
6
Current Data Religion Mystery Cult

• Data is shrouded in mystery
• Its meaning is only accessible through data
  priests
• Data Priests use omens (metadata) and personal
  experience to divine the meaning of data in
  return for (financial) sacrifices
• Meaning is often obscure, misleading, incomplete
  and wrong
• Data Priests are often blockers to data
  governance programs
• More paperwork
• Slows us down
• Dont need it

7
A Common Myth We know our data
Im a professional. Of course I know my data!

• Subject matter experts (SMEs) only know their own
  systems
• But they cant tell you how it changes and is
  transformed as it moves from system to system
• Relationships between systems are complex
• SMEs sometimes change jobs!

But, once it leaves my hands, it is someone
elses problem!
Wow, that transformation is complex. Are you
sure that is in my data?
Im going to start my own consulting firm
8
More of the Myth Our Data is Consistent
All of my data follows the business rules for
this system!

• Business rules are broken all the time as data
  crosses business and system boundaries
• 83 year old man in system A is a youthful
  driver in system B
• Bond yield is listed as 5 in system X and 5.3
  in system Y
• Exceptions result in lost revenue, customer
  dissatisfaction, and regulatory fines
• Business rules change as organizations change
• Mergers and Acquisitions
• New products or services
• Products/services are retired
• Reorganizations
• New IT systems are added

I cant keep up with all the acquisitions and
reorganizations. They mess up the way systems
work together. It is very inconvenient.
9
Data Ecosystem

• Warriors
• Have immediate, acute data problem
• Focus on feasibility and time

• Tradespeople
• Have a short term (project) business problem
• Focus on value

• Data Priests (SMEs)
• Know the data
• Assist with data problems

• Reformers
• Have a long term business problem
• Focus on control and scalability

10
Data Governance New Data Religion

• Governed Data
• Data is documented, consistent and secure
• Governance is a must for MDM projects
• To succeed, a new religion needs
• Reformers or prophets people who believe in it
  and sponsor it
• Priests people to educate, explain and promote
  it
• New Miracles successful projects and spectacular
  results

11
Successful Data Governance Program Roll Out
Help Warriors win battles
Use victories to convince Reformers to change
religions
Convert the priests
Have priests drive adoption to trades people
12
Key Winning the First Battle!

• Pick your battles
• Find appropriate initial project
• Achieve quick win
• Early success
• Build a Trojan Horse
• Govern the data without calling it that

13
Picking The First Battle

• Appropriate Project
• Immediate Business ROI
• Project success directly linked to Data
  Governance practice and methodology
• Cross-Silo
• Must have for the company
• Examples
• Basel II
• Master Data Management
• Application migration
• Cross-BU Reporting and Analytics

14
Avoid False Starts

• Projects to Avoid
• Future ROI next project will benefit
• Boil the ocean
• False Start Examples
• Refactoring
• Metadata repository
• Enterprise (fill in the blank)

15
Quick Win

• Iterative Approach
• Agile Development
• Immediate results
• Automation is Critical
• Data Discovery tools
• Repeatability of results
• Validation and consistency tools
• Incident/Exception workflow
• Visual and Intuitive Presentation of Results
• Business oriented
• Graphically presented

16
Barriers to the Quick Win
17
Data Governance Gap
The Peaks of Data Understanding
Design specifications get lost or outdated,
subject matter experts leave companies, databases
and business rules get changed without updating
documentation, mergers and acquisitions wreak
havoc on databases, all leading to a company not
knowing exactly what they have... The end result
is inconsistent data. Fern Halper, Hurwitz
Associates
Data Governance
Data Nightmare
70 or more of the time and effort involved in
completing most data integration projects is
consumed by defining and implementing the
business rules by which data will be mapped,
transformed, integrated, and cleansed. Ted
Friedman Vice President, Gartner Group
18
Current Tools Werent Developed to Discover a
Distributed Data Landscape

• ETL, EAI, Cleansing
• Not discovery solutions. They depend on
  discovery
• Metadata matching
• Doesnt work in a real environment
• Profiling
• Focused on a single data source
• Todays tools werent created to analyze a
  distributed data landscape
• Data analysts manually examine data values to
  figure out the business rules in the distributed
  data landscape
• The most sophisticated tools commonly used today
  is

Most Widely Used Business Rule Discovery Tools
19
Case Study Asset Master
Reminder for Todd Show Dawns slides
Vice President Charlotte, NC based Commercial bank

• Project
• IT Asset Master
• Consolidating 8 asset management systems to a
  single asset master

We had 9 subject matter experts spend 9
months and we still didnt know enough to be able
to consolidate our data into a master.
20
Data Analysis The Lack of UnderstandingA Case
Study(Note This is NOT what you want to do!)
21
Data Analyst Case Study

• The story you are about to hear is true
• Only the names have been changed to protect the
  innocent

22
This is Denise

• Experienced Data Analyst
• Extremely successful career working for data
  software companies
• Very Personable
• Very Intelligent
• Impeccable references
• Bills at 2000/day
• Hired by a dental insurance company for a 3 week
  data analysis/MDM integration project
• Tools used
• Profiling
• TOAD
• SQL
• Highlighter

Data Analyst Denise
23
Manual Data Discovery Timeline

• Get metadata specs and begin to check business
  rules between one table with six columns against
  first of three source systems
• Expected result
• 3 Weeks

Day 1
Data Analyst Denise
24
Manual Data Discovery Timeline

• Get initial results from unit test with
  inconsistent data for 1st column
• So far, so good

Day 2
Data Analyst Denise
25
Manual Data Discovery Timeline

• Retest and debug
• Still on track

Day 3
Data Analyst Denise
26
Manual Data Discovery Timeline

• Go to data architect to question 
• Architect pings owner of application (SME).
• NOTE  Data analyst not allowed to consult with
  SME directly.

Day 4-5
Data Analyst Denise
27
Manual Data Discovery Timeline

• Meeting with architect and SME to review. 
• Initial answer received .

Day 6
Data Analyst Denise
28
Manual Data Discovery Timeline

• Rewrite business rules and test. 
• Find second column with inconsistent data. 
• Retest and debug.

Day 7
Data Analyst Denise
29
Manual Data Discovery Timeline

• Go to data architect to question 
• Architect pings owner of application (SME).
• SME asks upstream application owner

Day 8-10
Data Analyst Denise
30
Manual Data Discovery Timeline

• Flurry of emails between the 4 players, as
  upstream app owner in different time zone. 
• Decision on how to proceed agreed upon

Data_Architect_at_company.com
Day 11-12
SME_at_company.com
App_Owner_at_company.com
Data Analyst Denise
App_Owner_at_company.com
App_Owner_at_company.com
denise_at_company.com
denise_at_company.com
denise_at_company.com
App_Owner_at_company.com
31
Manual Data Discovery Timeline

• Rewrite business rules in SQL and test. 
• Find more inconsistent data. 
• Retest and debug.

Day 13
Data Analyst Denise
32
Manual Data Discovery Timeline

• Go to data architect to question 
• Architect pings owner of application (SME).

Day 14
Data Analyst Denise
33
Manual Data Discovery Timeline

• Meeting with architect and SME to review. 
• Decision made to review specs with a larger group

Day 15
Data Analyst Denise
34
Manual Data Discovery Timeline

• Meeting with larger group. 
• Original specs validated and corrected

Day 16
Data Analyst Denise
35
Manual Data Discovery Timeline

• At weekly status meeting, project manager asks,
  why have 17 days passed when this phase was to
  be completed in 3 weeks?

Day 17
Data Analyst Denise
36
Manual Data Discovery Timeline

• Rewrite SQL and test.

Day 18
Data Analyst Denise
37
Manual Data Discovery Timeline

• Pass first source system SQL to ETL developers
  for coding and QA

Day 19
Data Analyst Denise
38
Manual Data Discovery Timeline

• Get specs and begin to verify relationships with
  second of three sources systems an outside feed

Day 20
Data Analyst Denise
39
Manual Data Discovery Timeline

• Go to data architect to question 
• Architect pings owner of application (SME).
• SME asks upstream application owner
• Feed vendor liaison is consulted

Day 21-23
Data Analyst Denise
40
Manual Data Discovery Timeline

• Flurry of emails between the 4 players, plus
  vendor liaison. 
• More people involved consumes even more time
• Decision on how to proceed agreed upon

Data_Architect_at_company.com
Feed_vendor_at_company.com
Day 24-26
SME_at_company.com
App_Owner_at_company.com
Data Analyst Denise
App_Owner_at_company.com
App_Owner_at_company.com
denise_at_company.com
denise_at_company.com
denise_at_company.com
App_Owner_at_company.com
41
Manual Data Discovery Timeline

• Recode SQL and test.
• Repeat experience of days 7-16, with new
  inconsistent data

Day 27
Data Analyst Denise
42
Manual Data Discovery Timeline

• Recode SQL and test.
• Repeat experience of days 7-16, with new
  inconsistent data

Day 37
Data Analyst Denise
43
Manual Data Discovery Timeline

• The project now 18 days overdue, with no clue as
  to how long it will take to complete the
  remaining work.
• Repeat variations of days 21-37 several times

Day 37
Data Analyst Denise
44
Manual Data Discovery Timeline

• Pass 2nd source system business rules to ETL
  developer and QA.
• Project phase is now 70 days overdue, with one
  entire source system still to code.
• Red flags being raised
• Search for sacrificial lambs.

Day 89
Data Analyst Denise
45
Manual Data Discovery Timeline

• Go on preplanned, and much overdue vacation

Day 90
46
Manual Data Discovery Timeline

• Get specs and begin to check business rules with
  third of three sources systems.
• Repeat variation of days 20-89.

Day 97
98
99
100
102
104
106
108
110
120
130
140
150
160
165
166
167
47
Manual Data Discovery Timeline

• Pass 3rd source system code to ETL developer and
  QA.
• Project is 152 days late
• 30 weeks
• 7 months
• Company paid for 30 weeks more consulting time
  than expected
• 300K overrun

Day
167
48
What does this mean for your Data Governance
Project?
49
MDM deployment10MM in Services for Every 1MM
in Software
Services

• MDM Hub
• Merge
• Purge
• Match

Software
50
According to the Experts
70 or more of the time and effort involved in
completing most data integration projects is
consumed by defining and implementing the
business rules by which data will be mapped,
transformed, integrated, and cleansed.
Ted Friedman Vice President Gartner Group
51
70 of Services are for Data Analysis30 of
Services Are for Deploying the Data Hub
Services

• Discover
• Map
• Validate

Data Analysis Services
MDM Deployment Services
Software
MDM Hub
52
According to the Experts
Malcolm Chisholm MDM Industry Expert AskGet, Inc.
MDM wont ever provide a positive return on
investment to businesses if the cost and risk of
the data analysis and mapping component is not
reduced by an order of magnitude you have to
automate the process
53
Recap So Far

• You must overcome big hurdles you must overcome
  to implement Data Governance

• Technical
• Presumes data understanding
• Requires automated data discovery, validation,
  remediation for a distributed data landscape

• Financial
• Cost of deployment must justify the project

• Cultural
• You may be changing religions

54
Data Governance Epic
The Peaks of Data Understanding
Data Governance
Data Nightmare
55
Data Governance Epic

• But you know the alternatives are unthinkable, so
  you and your team of data governance warriors
  boldly go where no man has gone before.

56
Data Governance Epic

• Scale the cliffs of data relationship discovery
• Pick your way through data inconsistency glaciers
• Battle Data Priests for budget and mindshare

57
Data Governance Epic

• And eventually, if you are very, very persistent
  and very, very lucky, you may even get there

58
Case Study Potential for the False Start

• Manufacturing Firm
• Corporate mandate to improve data quality (the
  CEO demanded a new religion)
• Created their initial identity master
• Initial Identity Master
• Required 5 analysts to map 4 data sources
• Merge purge match process is governed
• But Quality of data in the master is suspect
• Result No downstream users
• Next project
• 16 more sources to map
• Will require 20 more data analysts
• The Problem
• Hiring 20 data analysts is not financially
  feasible
• Data mapping and analysis is the critical path
• Millions of dollars have been spent on software
  and services already

59
Theres got to be an easier way!
Need a Quick Win
60
What if you could Automate Cross System Data
Understanding?
61
Automating cross system data discovery would
change the economics of governance from this
Services
MDM Deploy- ment Services
Software
MDM Hub
62
Automating cross system data discovery would
change the economics of governance from this
Services
MDM Deploy- ment Services
Software
MDM Hub
63
Automating cross system data discovery would
change the economics of governance to this

• Provides the foundation of good data management
• Automates understanding of the current data
  landscape
• Replace services with software (10x differential)
• Creates repeatability
• Makes data governance projects financially
  feasible
• Accelerate deployment
• Reduce project risk
• Turn negative NPV into positive NPV
• Provides the Trojan Horse

MDM Deploy- ment Services
Software
Analysis
MDM Hub
64
Case Study The Trojan Horse

• Truck Manufacturer
• Migrating from one finance application to another
• Data must be mapped and migrated as part of the
  process
• The Trojan Horse
• Some data in the finance application is master
  data
• Using automated tools to map the data and will
  leverage the map to create a master
• Did a pilot project where automation took 3 days
  vs 6 months for manual mapping
• Planned savings from automation are being
  rerouted to purchase an MDM system
• Critical Factors
• Governance processes will be required to clean up
  the data as part of the migration
• They are not calling this governance they are
  just doing it
• All mapping efforts will be leveragable because
  they are repeatable and verifiable
• Repeatable and verifiable are good words
• Future Challenges
• They must execute

65
Data Discovery Automation TechnologyA Primer
66
Automated Cross System Data DiscoveryWhat is it?

• New data analysis methodology and tools
• Arms the warrior with a new weapon
• Allows you to quickly understand your current
  data landscape
• Establishes data understanding within data
  sources and between data sources
• Automates discovery of business rules, lineage,
  transformations and data inconsistencies across
  data sources
• Goes well beyond profiling
• Examines
• Data Values
• Data Values
• Data Values
• Establishes a methodology for cross system data
  analysis
• Each data project becomes a building block, not a
  one-off

67
Data-Driven Approach Aligns Rows Across Datasets

• Step 1 Discovery Engine analyzes the data values
  to automatically discover the key that aligns
  rows across disparate datasets
• Works for hundreds of tables
• Works for millions of rows

68
Data-Driven Approach Aligns Rows Across Datasets

• Step 1 Discovery Engine analyzes the data values
  to automatically discover the key that aligns
  rows across disparate datasets
• Works for hundreds of tables
• Works for millions of rows

Table 1








69
Data-Driven Approach Discovers Business Rules
Sensitive Data

• Step 2 With rows now aligned, analyzes the data
  values to automatically discover
• Forgotten Business Rules
• Data Lineage
• Hidden Sensitive Data

Table 1








70
Data-Driven Approach Discovers Business Rules
Sensitive Data

• Step 3 With business rules now discovered,
  analyzes the data values to automatically
  discover
• Unknown Data Inconsistencies

Table 1








71
What Complex Business Rules are Discovered from
the Data?

• Scalar
• One to one
• Substring
• Concatenation
• Constants
• Tokens
• Conditional logic
• Case statements
• Equality/Inequality
• Null conditions
• In/Not In
• Conjunctions
• Joins
• Inner
• Left Outer

• Aggregation
• Sum
• Average
• Minimum
• Maximum
• Column Arithmetic
• Add
• Subtract
• Multiply
• Divide
• Reverse Pivot
• Cross-Reference
• Custom Data Rules

72
Case Study Worldwide Financial Institution

• Financial Services Firm
• Integration of legacy system with reference
  master system.
• First of 40 to be integrated
• Manual results for first dataset
• Estimated to take 6 months elapsed
• Data-Driven Mapping results
• 2.5 weeks of elapsed time
• Also centralized data analysis expertise
• Benefits
• Significant time to market savings 5 months
• Significant project risk reduction
• Data inconsistencies found as part of process

Master Data Management (Deployment time)
Months
73
What does this all mean?

• Makes it much easier and cheaper to map your
  distributed data landscape
• This is the foundation upon which the rest is
  built
• The economics of governance will look very
  different
• Faster, repeatable victories
• Turns point projects into governance building
  blocks
• Undoable projects become doable
• Turns data governance projects on their heads

74
Recap I
Culture use victories to build the case for
better data governance and quality
Trojan Horse Start governing your data without
calling it governance
Financial Use better data management to deliver
positive ROI
Technical Automate data discovery and management
75
One more point about Culture Change
Strategy
Communications
Organizational Structure
Culture
Training/Skills
Rewards
You cant just change culture. You have to turn
other knobs that affect culture
76
Recap II
77

Data Governance Success

• Happy CXO Management Team

78
Questions and Answers

• Thank You for Attending!
• For more information, contact
• Todd Goldman
• Web www.exeros.com
• Email todd_at_exeros.com
• Phone [phone]
• Or stop by Exeros Booth in the exhibit hall