TRLN High Performance Data Storage System 21 Sep 2006

1
TRLN High Performance Data Storage System21 Sep
2006

• Jim Porto
• Ken Galluppi

2
Not Many Answers

• We do not have a large disaster databaseno one
  really doesthats the problem.
• Data exists in many formats and is distributed
  across 100 counties and numerous state agencies.
• I think we are all open to using metadata tags,
  at a minimum, on the data that we do compile.

2
3
Not Many Answers

• We are open to standard metadata formats.
• Sizewe estimate that the size of all the
  databases that we want to compile will be on the
  order of 100 TB. Depends on what we add, but we
  could be adding up to 10 per year.
• A sizeable portion of the data could be permanent
  and will require long term storage. We hope to
  develop longitudinal data sets for time analysis.
• Dont know how much data will be generated per
  month or per year.

3
4
Not Many Answers

• Dont know yet either the granularity or the
  frequency of access.
• Software is not complex at this time, but
  development of very large database algorithms to
  tie spatial features together, to provide 3d
  spatial analysis, and to provide time series
  analysis for visual image displays and
  comparisons could become fairly complex.

4
5
(No Transcript)
6
Problems With NCOne Map DB

• Display onlycan not manipulate data once
  displayed.
• Distributed across 100 counties and multiple
  state agencies.
• Multiple formats for databases.
• Access and security concerns.
• Not all data is captured.

6
7
(No Transcript)
8
Characteristics of Database

• Could have a significant archive component.
• Could have a significant transient component
  requiring periodic flushing (ex. weather radar,)
• Builds relationships between permanent and
  transient data.
• One model is to link to distributed data bases
  (local, state, and federal), rapidly build
  transient databases both for mapping and for data
  manipulation, AND to partner with Triangle
  Universities to archive static data in data
  depositories.

8
9
Computer Science Issues

• How to maintain assembled data for transient use
  once maps have been rendered.
• Management of large data sets assembled on the
  fly.
• Time sampling and flushing of unused pieces.
• Data ownership.
• Data security.

9
10
Additional Computer Science Issues

• Data integrity compared to archived data.
• Bandwidth to move data around the network.
• Visualization of large datasets.
• How to build applications on top of large data
  sets (eg spatial analysis in standard GIS
  packages is not feasible)

10
11
Uses of Database

• Development of new modeling and display software
  that can handle 3d, time series, and very large
  databases.
• Substantive research into disaster
  vulnerabilities through large scale modeling.
• Source of regional and state operational
  information.
• Source of regional and state planning information.

11
12
Benefits of Database

• Aid to Disaster Decision Makers (General Assembly
  is increasingly interested in the States
  Disaster Response).
• Aid for Scenario construction and testing to
  improve preparedness.
• Cross discipline approach bringing in many
  professions. (Environmental quality, meteorology,
  public health, economics, planning and land use,
  agriculture, forestry, for example.)
• Regional economic analysis more feasible.

12
13
Benefits of Database

• Gain experience in working with ultra large
  databases.
• Develop new software tools to integrate , to
  visualize, and to analyze very large, diverse
  databases.
• Develop solutions for security and working
  agreements for data ownership.

13
14
Final Thoughts

• We want to advance the state of knowledge in the
  use if large datasets.
• We want to give the Triangle Universities a
  distinctive project that will garner national
  recognition.
• AND we want to provide relevant research that
  aids the citizens of NC (especially since this
  project if being funded by tax dollars.)

14