Managing the History of Metadata in support for DB Archiving and Schema Evolution

1
Managing the History of Metadata in support for
DB Archiving and Schema Evolution

• Carlo Curino (Politecnico di Milano)
• Hyun J. Moon (UCLA)
• Carlo Zaniolo (UCLA)

2
Outline

• Schema evolution in the real world
• Need for historical metadata management
• Our approach
• Other challenges of schema evolution
• Panta Rhei Framework
• Conclusion

3
Schema Evolution

• Changing environment force the DB to change
  (data and schema)
• We want to archive data and metadata histories
• Allow complex temporal queries over both data
  and metadata histories

4
Schema Evolution

• Schemas change. Why?
• Data reorganization
• Better logical design, performance
• Changes in the modeled reality
• New data collection, MA and restructuring,
  government regulations
• Is it real?
• Wikipedia DB 171 schema versions over 4.5 years
  ICEIS08
• More case studies Marche93 Sjoberg93

5
Case Studies
Wikipedia Number of Tables (4.5 years)
Wikipedia Number of Columns (4.5 years)
6
Metadata

• Metadata data about data
• Schema from information schema (schemata,
  tables, columns)
• Schema evolution Schema Modification Operators
  (SMOs) between schema versions
• Queries, also (historical) usage of data

7
Metadata History

• Metadata history management is needed
• Because schema keeps evolving
• Systematic archival and query support is needed
• What was the schema valid at 2005-07-01?
• What was the history of total column count?
• Whats the provenance of current column R.A?
• What queries (valid on a previous version) would
  fail after schema changes?

8
Archiving Metadata History

• V-document
• XML-based temporal data model VLDBJ09
• Temporally grouped attribute-level timestamping
• No redundancy, no need for coalescing
• Designed for data history archival
• We use it for metadata history archival

9
Querying Metadata History

• Retrieve the number of columns in the Wikipedia
  DB throughout the history

10
Querying Metadata History

• Whats the success rate of legacy queries (valid
  on 2004-08-15) after each Wikipedia schema
  changes?

11
Challenges of Schema Evolution

• Pain in DB migration and query adaptation
• Our approach PRISM workbench VLDB08a,ICDE09
• High-level language for schema modification,
  automatic data translation, quasi-automatic
  legacy query rewriting
• Archiving and querying of DB history made
  difficult
• Our approach PRIMA system VLDB08b
• Standard XML, query rewriting, optimizations
  using temporal semantics
• Management of schema history itself
• Our approach Historical Metadata Manager
  ECDM08

12
Panta Rhei Framework
13
Panta Rhei Resources

• Web Demos
• PRISM schema evolution workbench
• PRIMA historical data management
• Historical Metadata Manager (HMM)
• Homepage
• http//www.schemaevolution.org

14

• Thank you!

Question or Comments?
15
Extra Slides
16
Bibliography

• Clifford95 J. Clifford, A. Croker, F. Grandi,
  A. Tuzhilin. On temporal grouping. In Recent
  Advances in Temporal Databases, pages 194213.
  Springer Verlag, 1995.
• DeCastro97 C. De Castro, F. Grandi, Maria R.
  Scalas. Schema Versioning for Multitemporal
  Relational Databases. Inf. Syst. 22(5) 249-290
  (1997)
• ECDM08 C. A. Curino, H. J. Moon, C. Zaniolo.
  Managing the history of metadata in support for
  db archiving and schema evolution. In ECDM, 2008.
• ICEIS08 C. A. Curino, H. J. Moon, L. Tanca, C.
  Zaniolo. Schema Evolution in Wikipedia toward a
  Web Information System Benchmark, International
  Conference on Enterprise Information Systems
  (ICEIS) 2008
• ICDE09 C. A. Curino, H. J. Moon, M. Ham, C.
  Zaniolo, The PRISM Workbench Database Schema
  Evolution Without Tears. To Appear in ICDE 2009
  (Demo)
• Marche93 S. Marche. Measuring the stability of
  data models, European Journal of Information
  Systems, 2(1)37-47, 1993.
• Roddick95 J. Roddick. A Survey of Schema
  Versioning Issues for Database Systems.
  Information and Software Technology,
  37(7)383393, 1995.
• Sjoberg93 D. I. Sjoberg. Quantifying schema
  evolution, Information and Software Technology,
  35(1)35-44, 1993.
• VLDB08a C. A. Curino, H. J. Moon, C. Zaniolo.
  "Graceful database schema evolution the PRISM
  workbench". VLDB, 2008.
• VLDB08b H. J. Moon, C. A. Curino, A. Deutsch,
  C.-Y. Hou, C. Zaniolo. "Managing and querying
  transaction-time databases under schema
  evolution". VLDB, 2008.
• VLDBJ09 F. Wang, C. Zaniolo, X. Zhou. ArchIS
  An XML-based approach to transaction-time
  temporal database systems Accepted for
  publication in VLDB Journal.