This presentation contains information proprietary to Oracle

1
(No Transcript)
2
Oracle Data Integrator Technical Deck

• Mark Pare
• Sr. Sales Consultant
• Oracle Higher Education

3
Agenda
ltInsert Picture Heregt

• 4 Key Differentiators
• Architecture
• 6 Steps to Production
• ODI or ESB?
• Popular Usage Scenarios

4
Oracle Data Integrator 4 Key Differentiators
5
Why Data Integration?
NEED
HAVE
6
Challenges Emerging SolutionsIn Data
Integration
CHALLENGE
EMERGING SOLUTION

• Increasing data volumes decreasing batch windows
• Non-integrated integration
• Complexity, manual effort of conventional ETL
  design
• Lack of knowledge capture

Shift from E-T-L to E-LT Convergence of
integration solutions Shift from custom coding
to declarative design Shift to pattern-driven
development
7
Oracle Data Integrator

• Data Movement and Transformation from Multiple
  Sources to Heterogeneous Targets
• Performance Heterogeneous E-LT
• Flexibility Active Integration Platform
• Productivity Declarative Design
• Hot-Pluggable Knowledge Modules

BENEFITS
KEY DIFFERENTIATED FEATURES
8
Differentiator E-LT ArchitectureHigh Performance
1

• Transform in Separate ETL Server
• Proprietary Engine
• Poor Performance
• High Costs
• IBM Informaticas approach
• Transform in Existing RDBMS
• Leverage Resources
• Efficient
• High Performance
• Benefits
• Optimal Performance Scalability
• Easier to Manage Lower Cost

9
Differentiator Active IntegrationBatch,
Event-based, and Service-oriented Integration
2

• Evolve from Batch to Near Real-time Warehousing
  on Common Platform
• Unify the Silos of Data Integration
• Data Integrity on the Fly
• Services Plug into Oracle SOA Suite
• Benefits

• Enables real-time data warehousing and
  operational data hubs
• Services plug into Oracle SOA Suite for
  comprehensive integration

10
Differentiator Declarative DesignDeveloper
Productivity
3

• Specify ETL Data Flow Graph
• Developer must define every step of Complex ETL
  Flow Logic
• Traditional approach requires specialized ETL
  skills
• And significant development and maintenance
  efforts
• Declarative Set-based Design
• Simplifies the number of steps
• Automatically generates the Data Flow whatever
  the sources and target DB
• Example SALAH
• Benefits
• Significantly reduce the learning curve
• Shorter implementation times
• Streamline access to non-IT pros

11
Differentiator Knowledge ModulesHot-Pluggable
Modular, Flexible, Extensible
4
Pluggable Knowledge Modules Architecture
Journalize Read from CDC Source
Load From Sources to Staging
Check Constraints before Load
Integrate Transform and Move to Targets
Service Expose Data and Transformation Services
Reverse Engineer Metadata
Reverse
WS
WS
WS
Staging Tables
Load
Integrate
Services
CDC
Check
Journalize
Target Tables
Sources
Error Tables
Sample out-of-the-box Knowledge Modules
SAP/R3
Log Miner
SQL Server Triggers
Oracle DBLink
JMS Queues
Check MS Excel
TPump/ Multiload
Oracle Merge
Oracle Web Services
Siebel
DB2 Journals
DB2 Exp/Imp
Check Sybase
Oracle SQLLoader
Type II SCD
Siebel EIM Schema
DB2 Web Services
Benefits

• Tailor to existing best practices
• Ease administration work
• Reduce cost of ownership

12
Oracle Data IntegratorArchitecture
13
Architecture Conceptual View

• Java design-time environment
• Runs on any platform
• Thin client for browsing Metadata
• Java runtime environment
• Runs on any platform
• Orchestrates the execution of data flows
• Metadata repository
• Pluggable on many RDBMS
• Ready for deployment
• Modular and extensible metadata

14
Architecture Component View
15
Oracle Data Integrator6 steps to Production
16
Overview 6 steps to Production

• Retrieve/Enrich metadata
• Design transformations
• Orchestrate data flows

• Generate/Deploy data flows
• Monitor executions
• Analyze impact / data lineage

Production
Development
Production Servers and Applications
Development Servers and Applications
Data Warehouse
Data Warehouse
CRM
CRM
Legacy
Legacy
ERP
ERP
ODI Design-Time Environment
ODI Runtime Environment
Agent Data Flow Conductor
Agent Data Flow Conductor
Operator
Metadata Navigator
17
Retrieve/Enrich Metadata
1
Design-Time Environment
Development Servers and Applications

• Reverse-engineer Metadata
• Automatic
• Customizable
• 40 technologies supported
• Enrich Metadata
• Documentation
• Declarative rules for Data Integrity
• Cross-technologies references

ODI Designer
ERP
Data Warehouse
Design-time Repositories
CRM
Files / XML
Legacy
18
Design Transformations
2
1
Define What You Want
Oracle Data Integrator Interface Declarative
Design
Bulk Load Changed Data Capture
Incremental Update Slowly Changing Dimension
19
Orchestrate Data Flows
3

• Sequence Transformations
• Leverage OracleDI Tools
• Data Quality Processes
• Files/Archives Management
• Send/Receive Emails
• Web Services Invokation
• Event Detection
• Create your Own Tools
• Use Control Structures
• Loops
• Conditions
• Error Handling

20
Generate and Deploy Data Flows
4

• Create Scenarios
• Compile Data Flows for Run-time
• Version the Data Flows
• Advanced Version Management
• Deploy to Production

21
Monitor Executions
5

• View sessions running in real-time
• Review generated code
• Detailed run-time statistics
• Restart failed sessions

22
Analyze impact / data lineage
6

• Maintain a large number of data flows in a
  complex environment
• Web-based end-to-end data lineage
• Understand your data flows
• Follow the path of data
• Drill-down to transformations

?
23
Oracle Data IntegratorODI or ESB?
24
What tool is best suited for task X?
25
ESB and ODI in real-life scenarios
Data Latency
Oracle Data Integrator
Batch (over 2 hours)
Oracle Enterprise Service Bus
Asynchronous
Synchronous (immediate)
Real-life Scenarios
Data Volume Processing
Message by Message
Mini Batches
Large Volume (over 1M)
26
Oracle Data IntegratorExtended Capabilities
27
Extended Capabilities

• Master Data Management enabled
• Common Format Designer
• Automated generation of canonical format and
  transformations
• Built-in Data Integrity
• Real-time enabled
• Changed Data Capture
• Message Oriented Integration (JMS)
• SOA enabled
• Generation of Data Services
• Generation of Transformation Services
• Extensibility
• Knowledge Modules Framework
• Scripting Languages
• Open Tools

28
Oracle Data IntegratorMaster Data Management
Enabled
29
MDM Enabled Canonical Format Design

• Use in conjunction with packaged MDM solution
• Design and Populate Canonical Format
• Use existing metadata artifacts to design MDM
  application (entities, fields, relationships)
• Generate and maintain Master Data structure
• Generate and deploy transformations using
  metadata artifacts

Enterprise Service Bus
CRM
SCM
Legacy
ERP
30
MDM Enabled Built-in Data Integrity

• Data Integrity Firewall
• Auditing, cleansing and recycling
• Declare constraints at table level
• Design mappings and check flow integrity
• Audit, cleanse or recycle rejected records

31
Oracle Data IntegratorReal-time Enabled
32
Real-time enabled Changed Data Capture

• Publish and Subscribe CDC Framework
• Database logs
• Triggers
• Third-tier solutions
• Ensures read transaction integrity across
  multiple tables
• Design or generate Mappings
• Select Journalized Data Only
• Start Journals

CDC
33
Real-time enabled Message Oriented Integration

• Connect to Publish and Subscribe JMS Message
  Providers
• Ensure messages delivery with transaction
  integrity
• High-volume bulk transformations
• Design complex bulk transformations mixing
  Queues, Databases and Applications
• Use JMS Queues and topics as sources or targets

CDC
34
Oracle Data IntegratorSOA Enabled
35
SOA Enabled Data Access Services
SOA Infrastructure

• Generate and share data access services
• Generate and deploy data services
• Test data services
• Leverage data services in your SOA infrastructure

Business Processes
Services
Data Access
Transform
ESB
Business
36
SOA Enabled Data Flow Services

• Expose transformations as Web Services
• Orchestrate data flows
• Publish data flows as web services in your SOA
  infrastructure

SOA Infrastructure
Business Processes
Services
Data Access
Transform
ESB
Business
37
Oracle Data IntegratorExtensible Framework
38
Extensibility Knowledge Modules

• 120 KMs out-of-the-box
• Tailor to existing best practices
• Ease administration work
• Reduce cost of ownership
• Customizable and extensible

KM Interpreter
39
Extensibility Scripting Framework

• Extend data flows with scripting procedures
• Leverage all database languages
• SQL, PL/SQL, Transact SQL, etc.
• Use Operating Systems shell scripts
• Win32 DOS, sh, ksh, csh, OS400 commands, JCL,
  etc.
• Choose from compatible Bean Scripting Framework
  languages
• Java, JavaScript, Jython (Java Python), Perl,
  etc.

40
Extensibility Open Tools

• Extend ODI tools
• Add your own tools to the Design Palette
• Implement OdiOpenToolAbstract Java Interface
• Register Open Tool in ODI Designer
• Use Open Tool in your design environment

41
Popular Usage Scenarios
42
E-LT for Data WarehouseCreate Data Warehouse for
Business Intelligence Populate Warehouse with
High Performance ODI

• Heterogeneous sources and targets
• Incremental load
• Slowly changing dimensions
• Data integrity and consistency
• Changed data capture
• Data lineage

43
SOA InitiativeEstablish Messaging Architecture
for Integration Incorporate Efficient Bulk Data
Processing with ODI

• Invoke external services for data integration
• Deploy data services
• Deploy transformation services
• Integrate data and transformation services in
  your SOA infrastructure

44
Master Data ManagementCreate Single View of the
TruthSynchronize Data with ODI

• Use in conjunction with packaged MDM solution
• Use as infrastructure for designing your own hub
• Create declarative data flows
• Capture changes (CDC)
• Reconcile and cleanse the data
• Publish and share master data
• Extend metadata definitions

45
MigrationUpgrade Applications or Migrate to New
Schema Move Bulk Data Once and Keep in Sync with
ODI

• Bulk-load historical data to new application
• Transform source format to target
• Synchronize new and old applications during
  overlap time
• Capture changes in a bi-directional way (CDC)

46
ODI Enhances Oracle BIPopulate Warehouse with
High Performance ODI

• Oracle Business Intelligence Suite EE
• Simplified Business Model View
• Advanced Calculation Integration Engine
• Intelligent Request Generation
• Optimized Data Access
• Oracle Data Integrator
• Populate Enterprise Data Warehouse
• Optimized Performance for Load and Transform
• Extensible Pre-packaged E-LT Content

47
ODI Enhances Oracle SOA SuiteAdd Bulk Data
Transformation to BPEL Process

• Oracle SOA Suite
• BPEL Process Manager for Business Process
  Orchestration
• Oracle Data Integrator
• Efficient Bulk Data Processing as Part of
  Business Process
• Interact via Data Services and Transformation
  Services

48
ODI Enhances Oracle SOA SuitePopulate BAM Active
Data Cache Efficiently

• Oracle SOA Suite
• Business Activity Monitoring for Real-time
  Insight
• Oracle Data Integrator
• High Performance Loading of BAMs Active Data
  Cache
• Pre-built and Integrated

49
Roadmap and Direction
50
Oracle Data Integrator Roadmap

• Focus Areas for Next Major Release
• Deep Integration with Fusion Middleware
• Runtime, Design time, Security, Administration,
  Events
• Functional Integration with Oracle Warehouse
  Builder
• Runtime Integration, Metadata Sharing, Knowledge
  Module Sharing
• Deployment of ODI for Embedded Data Integration
• OracleBI Enterprise Edition, Data Hubs,
  Application Migrations
• Enhanced Usability and Debuggability
• Wizards, New Views, User-definable Debugging
• Improved Support for Native Oracle Database
  Features
• Oracle OLAP

51
ODI Statement of Direction

• Statement of Direction
• http//www.oracle.com/technology/products/odi/stat
  ement-of-direction.pdf
• Key Points of Direction
• Commitment to heterogeneous systems support
• Including DB2, Teradata, Netezza, Hyperion,
  etc.
• Commitment to Fusion design principles
• Including J2EE compliance, container
  portability
• Commitment to best-of-class E-LT performance
• Across platforms, batch realtime, high
  complexity

52
(No Transcript)
53
(No Transcript)