MWCOG GIS Committee Meeting Regional Geospatial Interoperability

1
MWCOG GIS Committee MeetingRegional Geospatial
Interoperability
May 25, 2006 Rich Grady, President Applied
Geographics, Inc.
2
Introduction

• Applied Geographics, Inc. (AppGeo)
• www.appgeo.com
• GIS Consulting firm in Boston,15 years in
  operation, 25 professionals
• Service-provider in several Homeland Security and
  Emergency Management focused GIS projects
• Focused on State Local Government GIS system
  planning and implementation

3
General Observations on Homeland Security

• There are many stakeholders involved in Homeland
  Security, representing many perspectives.
• Stakeholders include 
• Different levels of government (federal, state,
  and local), and the private sector
• Different professional perspectives, including,
  public works, public safety, public health,
  planning, and GIS
• It is essential for a region to communicate and
  coordinate on a proactive basis

4
Questions to think about

• What data is needed?
• Who has it?
• How do we achieve interoperability?
• Are there some good examples of whats been done
  to-date?

5
Considerations

• Inventory of Available Data
• Application Objectives
• Standards
• Schema Design
• Data Collection and Integration
• Data Maintenance
• Metadata
• Output
• Security

6
Many Different Sources ( schemas, standards,
products, applications, reference systems,
etc.)!!!
Regional
Municipal
Data Repository
Private
County
Military
State
Federal
Data Sources wide variety of quantity and quality
7
Project Examples

• Overview of Boston Preparedness Pilot
• Defined First Responder requirements
• Identified data integration issues
• Observations from NCR/HSIP Pilot and involvement
  in Project Homeland HSIP refinement
• Issues of schema development and normalization
• Issues of practical applications

8
The Boston Preparedness Pilot Project, Sept. 2003

• Cover story of GeoIntelligence Magazine Mar/Apr
  04
• http//www.appgeo.com/clients/NOAA_HomelandSecurit
  y
• Phase I Funded by MassGIS
• Poll First Responders for GIS data needs
• Explore data schema design issues
• Phase II Funded by NOAA
• Develop/assemble priority critical infrastructure
  (CI) data layers
• Offer lessons learned to other 132 Urban Areas

9
Pilot Area 102 Cities Towns
10
Boston Preparedness Pilot Project Data Quality
Recommendations

• Assembling the most accurate data requires local
  input
• Local data must rollup to national level, not
  vice versa
• Assembling the data is only the first step,
  mechanisms for maintaining the data are essential
• Building high-quality homeland security and
  critical infrastructure protection applications
  implies a need for laborious data quality
  improvement work
• Unglamorous data development and data quality
  enhancement work is a necessary precursor to more
  sexy application development work
• On-line, web-based markup tools can greatly
  facilitate the ability for widely decentralized
  organizations to participate in data collection
  and quality assurance efforts

11
Boston Preparedness Pilot Project
RecommendationsOn the role of the Federal
Government

• Recommendation that local data should rollup
  into national data sets and that the Federal role
  in direct data collection might be limited
• Purchasing imagery
• Establishing blanket agreements with commercial
  data providers
• However, the Federal Government plays a vital
  role to help orchestrate decentralized data
  collection efforts
• Setting standards for data format and content
• Local transactions, National Specifications
• Setting ground rules for data collection
  practices
• Facilitating the development of the recipe that
  each Urban Area should follow

12
Homeland Security Infrastructure Protection
(HSIP) / National Capital Region (HSIP/NCR) Pilot
Project

• Worked on project for USACE and NGA as sub to
  Michael Baker Corp. during 2004
• Explored data schema options and complexities and
  piloted development for NCR
• Reconcile SDSFIE with HSIP
• Develop a schema for a subset of HSIP and
  implement in a personal Geodatabase
• HSIP layer list vs. structured schema
• Conceptualize HSIP data warehousing scenarios
  and ETL procedures
• Develop tools for schema-to-schema mapping and
  metadata tracking from multiple sources
• Populate target HSIP schema with actual data from
  NCR
• Document lessons-learned and recommendations

13
Project Overview

• Review of prevalent schema literature
• HSIP Schema formalization and SDSFIE, FACC
  crosswalks
• GIS Data Warehouse Processes and Software Tools
• National Capitol Region Pilot

14
Public State and Local GIS Data

• Area of interest 3 states and 8 counties
• Required data request letter from Federal
  customer
• Initial web-research on local data/metadata
  catalogs
• Data Use and License Agreements required
• Primarily planimetric data and base mapping
• Lacking in energy/gas/water utilities and
  telecommunications

15
National Capitol Region
16
6 Phase Dataset Workflow

• Data Inventorying and Acquisition
• Data Registration, Geometric Transformation, and
  Spatial QA/QC
• Schema Mapping and Tabular Transformation
• QA/QC, Completeness, Supplemental Data Activity
• Data Load to Central Repository
• Data Distribution and Use

17
Phase 1 Data Inventorying and Acquisition
SDSFIE
Federal
Military
EPA Tier II
Dataset Tracker
Inventory Acquisition
FACC
GDT
Commercial
Local
County
18
Phase 2 Data Registration, Geometric
Transformation, and Spatial QA/QC
SDSFIE
Geom Transform
FME
Persistent Tabular Transform
EPA Tier II
Maintain Original Primary Key
FACC
Staging Dataset 1
GDT
Spatial QA/QC
Staging Dataset 2
County
ArcMap
19
Phase 3 Schema Mapping and Tabular Data
Transformation
Staging Dataset Schema
Schema Map
Mapper
Tabular Schema Mapping
Target HSIP Schema
Staging Dataset 2
Data Complete Check
Tabular Transform
DTS
Staging Dataset 3
Persistent Tabular Transform
20
Phase 4 QA/QC, Completeness, and Supplemental
Data Activity
Data Complete Check
Tabular Schema Mapping
Staging Dataset 3
Data Collection
Internet, Misc.
Staging Dataset 4
21
Phase 5 Data Load to Central Repository
Merge To Central DB
Staging Dataset 4
Central Repository Dataset 1
ArcGIS DTS
Central Repository Dataset 2
22
Phase 6 Data Distribution and Use
Distribute
Data Access and Products
Central Repository Dataset
Desktop or Web Use
Return Improved Data to Provider
23
Warehouse Process and Software Tools Cross
Reference
24
Warehouse Software Tools

• ESRI ArcGIS ArcMap/ArcEditor (COTS)
• SafeSoftware Feature Manipulation Engine (FME)
  (COTS)
• Microsoft Data Transformation Services (DTS)
  (COTS)
• Data Warehouse Dataset Tracker Application
• Schema Mapper Application
• Data Review and Completeness Checker
• HSIP Schema Browser
• HSIP Schema Downloader

25
Project Homeland

• Worked under ESRI on contract to National
  Geospatial-Intelligence Agency (NGA) during
  2004-2005
• Developed a refined Homeland Security
  Infrastructure Protection (HSIP) layer list,
  including Sectors and Features, with associated
  documentation
• Built a schema for the refined HSIP
• Implemented in a personal Geodatabase with
  associated documentation
• Delivered to ESRI for application to Pilot Cities
  (e.g. San Francisco, Colorado Springs, Arizona
  Border Area)

26
Findings across pilot projects

• Schema issues are important but have yet to be
  resolved categorically
• Project homeland is ongoing
• People continue to wait for the next HSIP
• Simplicity aids in data warehousing and
  interoperability
• Complexity aids in rich applications
• Highlights importance of developing these schemas
• Data contributors want the roadmap

27
Complexity Range Schemas, Data and Applications
Custom Structures/Applications
Statistical Models
Machine Processing
CATS
EPA Tier 2
HAZUS
SDSFIE
Normalized
Network Tracing
Application Sophistication
ESRI HS
HSIP
GeoDatabase
De-normalized
FACC
ArcIms ShapeFile
Human Visual
Basic Data
Features Attributes Domains Relationships
Cost
28
Interoperability can benefit from schema
simplification

• Easier to exchange and import into central
  repository
• The ETL is easier (extraction translation
  loading)
• Facilitates use of XML
• Easier to QAQC and cleanup
• Easier to use in viewing applications

29
Tabular attribute correlations and
transformations the hardest part of the job
By separating geometry from attributes, we have
a wider selection of ETL tools to choose from for
processing tabular data.
30
Distributed responsibilities and a common
schema for interoperability are needed

• The authoritative data sources and management
  operations are distributed.
• The central warehouse authority reviews and helps
  coordinate data gathering, but does not actually
  gather or QAQC data.
• Given a common schema, data can be rolled up and
  processed with the same queries and applications.
  

31
Geocoding is not good enough
Actual Building Location
Geocoded Source A (MassGIS)
Geocoded Source B Commercial Vendor
32
Data Accuracy Refinement
1. Geocoded Point
2. Visual or Locally Assisted Confirmation of
Asset
3. Footprint-level Feature Capture
33
Multi-role facilities need to be handled
Maintain Feature Location 24 Bigelow
St.Contact Info.
Geocode
Facility Records Management
Feature Records Management
Maintain Map Label
Maintain Map Label
Maintain Map Label
Police Station
Mass Care Shelter
Fire Station
34
First Responders also require extensive, reliable
attributes

• Contact Info
• One structure may have multiple contacts
• Owner
• Superintendent
• Tenants
• Addresses
• Phone numbers

• Context Info
• Where is the building and what is nearby?
• Is the building a
• School
• Police station
• Hospital
• Nursing home/elder care
• Shelter
• Staging area
• High risk terrorist target
• Etc.
• One building can have multiple contexts

35
Summary We need more than GIS Software for
Interoperability

• Coordination between stakeholders
• Cooperative agreements for data sharing and
  interoperability
• Data warehousing approaches and ETL tools
• Emergency response applicationsIn the past, GIS
  products often contained custom built databases
• A practical and authorative HSIP schema
• Better attribute integration
• National standards with local implementation

36
Questions Discussion

• Thank you
• Rich Grady
• grady_at_appgeo.com