NIEM and Developing Exchange Standards

1
NIEM and Developing Exchange Standards
2
SEARCH

• The National Consortium for Justice Information
  and Statistics
• SEARCH is a non-profit membership organization by
  and for the 50 states and territories
• Organized in 1969
• Members are Governor-appointed justice and public
  safety professionals who guide the organization
  and affect policy on a national level
• Bureau of Justice Assistance, U.S. Department of
  Justice funds SEARCH to provide technical
  assistance for information sharing

3
National Information Exchange Model (NIEM)

• Partnership between U.S. DOJ and DHS (soon ODNI)
• Designed to develop, disseminate and support
  enterprise-wide information exchange standards
  and processes
• Enable jurisdictions to share critical
  information in emergency situations
• Support day-to-day operations of agencies
  throughout the nation

4
NIEM Governance

• NIEM Executive Steering Committee
• Policy Advisory Panel
• NIEM Project Management Office
• NIEM Business Architecture Committee
• NIEM Technical Architecture Committee

5
What is the NIEM?

• Object-oriented data model
• Instantiated as XML Schema
• Composed of reusable components
• Designed to facilitate disparate government and
  private entities in exchanging information
  quickly, accurately, and reliably

6
What is the NIEM?

• A framework that can be adapted/extended by
  local, state, tribal, and federal jurisdictions
  to support information sharing
• A work in progress
• Used to develop exchange specifications

7
NIEM Really is

• A mechanism to facilitate the standardization of
  data exchanged between disparate systems

8
NIEM Subject Areas
Infrastructure Protection
Immigration
Intelligence
International Trade
Screening
Criminal Justice
Emergency Management
9
XML Namespaces

• NIEM core and NIEM domains reside in own
  namespace

10
NIEM consists of

• Number of types 777
• Universal 106
• Common 106
• Domains 565
• Number of properties 3895
• Universal 513
• Common 953
• Domains 2429

11
What does NIEM help accomplish?
Replace this
with this
Individual interfaces
One standard
12
What is an IEPD?

• Information Exchange Package (IEP)
• Represents a set of data that is transmitted for
  a specific business purpose. 
• It is the actual XML instance that delivers the
  payload or information.
• Information Exchange Package Documentation
• (IEPD or IEP Documentation)
• Is a collection of artifacts that describe
  structure content of an Information Exchange
  Package.
• It does not specify other interface layers (such
  as Web services)

13
Analogy

• NIEM is like Websters Dictionary standard
  spellings and definitions of words
• Exchange specification (IEPD) is like a novel
  brings together specific words to provide
  contextual meaning
• Reading the dictionary isnt very interesting!
• NIEM novels are called Information Exchange
  Package Documentation (IEPD)

14
Reference IEPD

• Base exchange template that should be reused
  as-is or modified to perform a similar business
  function
• Endorsed by an Authoritative Source
• Many found in IEPD clearinghouse

15
IEPD Scope
Data mapped to published NIEM schema
NIEM IEPD Scope
Import / Export Tool
Import / Export Tool
DATA
Data mapped to local database - unpublished
DATA
DATA
Data mapped to local database - unpublished
Database 1
Database 2
16
IEPD Process
17
Work Group

• IEPD Development cannot be done by
• Technical experts alone
• Business experts alone
• Should be a partnership between developers and
  business
• Include necessary stakeholders from all sides of
  the exchange
• Skilled, experienced facilitator is important
• Ideal size dependant on initiative

18
NIEM Mechanisms

• Associations
• Roles
• Metadata
• Augmentation

19
Associations

• Relationship between autonomous components
• Accommodates
• N-Ary component association
• Association properties

20
NIEM Association Structure (partial)
Disparate entities tied together by the
references of the association
The type of association
Properties unique to the association
21
Associations Example

• Lisa Simpson and Bart Simpson are siblings.

22
Associations Example

• ltncImmediateFamilyAssociationgt
• ltncPersonChildReference ref"A1"/gt
• ltncPersonChildReference ref"B1"/gt
• ltncFamilyKinshipTextgtSiblingslt/ncFamilyKinshi
  pTextgt
• lt/ncImmediateFamilyAssociation gt
• ltncPerson id"A1" gt
• ltncPersonNamegt
• ltncPersonGivenNamegtLisalt/ncPersonGivenNamegt
• ltncPersonSurNamegtSimpsonlt/ncPersonSurNamegt
• lt/ncPersonNamegt
• lt/ncPersongt
• ltncPerson id"B1" gt
• ltncPersonNamegt
• ltncPersonGivenNamegtBartlt/ncPersonGivenNamegt
• ltncPersonSurNamegtSimpsonlt/ncPersonSurNamegt
• lt/ncPersonNamegt
• lt/ncPersongt

23
Roles

• Represents a particular function for an entity
• Some examples relating to people
• ICE Officer
• Pedestrian
• Weapon
• Roles also apply to property, organizations and
  others
• A single entity can take on many roles

24
Role Structure

• Entities considered roles are stand-alone data
  types (e.g., AlienType,VictimType, EquipmentType)
• Contains one of the following to associate the
  role function to base object
• RoleOfPerson
• RoleOfOrganization, or
• RoleOfProperty
• References the relevant target at run-time

25
RoleOf Example

• Chief Clancy Wiggum slipped on a bar of soap
  thrown by Bart Simpson when responding to a
  service call by Apu Nahasapeemapetilon for trying
  to steal a candy bar from the Kwik-E-Mart. He
  bruised his Coccyx.

VictimInjury
IncidentType
has-a
PersonType
VictimType
RoleOf
has-a
EnforcementOfficialType
RoleOf
has-a
BadgeID
26
Role Example XML Instance

• ltncPerson sid"P1"gt
• ltncPersonNamegt
• ltncPersonGivenNamegtClancylt/ncPersonGivenNamegt
• ltncPersonSurNamegtWiggumlt/ncPersonSurNamegt
• lt/ncPersonNamegt
• ltncPersonBirthDategt1960-10-07lt/ncPersonBirthDat
  egt
• lt/ncPersongt
• ltjIncident sid"I1"gt
• ltjIncidentResponseOfficial sid"RO1"/gt
• ltncRoleOfPersonReference sref"P1"/gt
• ltjEnforcementOfficialBadgeIDgt
• ltncIDgt421lt/ncIDgt
• lt/jEnforcementOfficialBadgeIDgt
• lt/jIncidentResponseOfficialgt
• ltjIncidentVictim sid"V1"gt
• ltncRoleOfPersonReference sref"P1"/gt
• ltjVictimInjurygt
• ltcInjuryTypeTextgtBruised Coccyxlt/jIDTypegt
• lt/jVictimInjurygt

27
Metadata

• Is data about data, or data about information
• Typically captures info on how to
• Retrieve, navigate, etc.
• e.g., topical information such as name, class, or
  category
• Assess data pedigree or origin
• e.g., accuracy, precision, reliability, belief,
  timeliness
• Is context dependent!
• Data in one exchange may be metadata in another

28
Metadata Data Model Structure
Structures namespace only defines linkages
Some universally applicable metadata
All types with an is-a relationship with
ComplexObjectType can use the linkages
29
Metadata Instance Example

• ltncPerson smetadata"M1 M2"gt
• ltncPersonNamegt
• ltncPersonGivenNamegtAdamlt/ncPersonGivenNamegt
• ltncPersonSurNamegtBrookslt/ncPersonSurNamegt
• lt/ncPersonNamegt
• ltncPersonBirthDategt1960-10-07lt/ncPersonBirthDa
  tegt
• lt/ncPersongt
• ltncMetadata sid"M1"gt
• ltncReportedDategt2005-08-01lt/ncReportedDategt
• lt/ncMetadatagt
• ltlocal-nsMetadata sid"M2"gt
• ltlocal-nsDatabaseIDgt2829019291
• lt/local-nsDatabaseIDgt
• lt/local-nsMetadatagt

30
Type Augmentation

• Groups domain elements into Augmentation groups
• Augmentation groups are implemented using
  substitution groups (xml)
• Objects in an IEPD can then extend base types by
  pulling in Augmentations from multiple domains
• More like a Person with a bag of Emergency
  Management things and a bag of Justice things
• Less like a special Emergency Management Person
  or Justice Person
• Extended base types are done using substitution
  groups

31
Type Augmentation Purpose

• To enable reuse of type extensions that occur
  within one domain to be used in another
• Minimizes duplicate extensions in multiple
  domains
• In Object Oriented terms, it is a means of
  addressing the multiple inheritance problem

32
NIEM Structure for PersonAugmentation
sAugmentation
Head element for substitution group
member-of
jPersonAugmentation
jPersonAugmentationType
of-type
intelPersonAugmentation
intelPersonAugmentationType
of-type
Remaining Domains
Substitutable elements
33
Specific Example IEPD Schema
ltcomplexType name"IEPDPersonType"gt
ltcomplexContentgt ltextension
basencPersonType"gt ltsequencegt
ltgroup ref"jPersonAugmentation"/gt
ltgroup ref"intelPersonAugmentation"/gt
lt/sequencegt lt/extensiongt lt/complexContentgt lt
/complexTypegt ltelement name"IEPDPerson
typelocal-nsIEPDPersonType substitutionGrou
pncPerson /gt
Makes use of implied substitution groups as
described earlier
34
Specific Example IEPD Instance
ltiepdIEPDPersongt ltncPersonBuildTextgtHuskylt/nc
PersonBuildTextgt lt!-- More Common
Person-related elements --gt ltjPersonAugmentatio
ngt ltjPersonDrivingInsuranceStatusTextgt None
at all lt/jPersonDrivingInsuranceStatusTextgt
lt!-- More Justice Person-related elements --gt
lt/jPersonAugmentationgt ltintelPersonAugmentat
iongt ltintelOccupationNamegtNIEM
Trainerlt/intelOccupationNamegt lt!-- More
Intel Person-related elements --gt
lt/intelPersonAugmentationgt lt/iepdIEPDPersongt
35
IEPD Process
36
Exchange Content Modeling

• In building an IEP, it is critical to have
• Precise definition/description of exchange
  structure
• Not limited paper documents
• Description that can be understandable and
  verifiable by all stakeholders (bridge the
  communication gap)
• Description technique that facilitates
  interactive design

37
Gathering Exchange Requirements

• Existing Documents
• Existing Business Requirements
• Stakeholder Knowledge

38
NIEM as Source of Exchange Concepts

• NIEM contains many agreed upon concepts and
  mechanisms
• Use these if they fitdo not reinvent the wheel
• Do not use them if they do not fitdo not
  restrict your exchange model to what is in NIEM
• Build the exchange model to reflect your business
• Build a model that the business people can
  understand and agree to

39
Best Practices

• Life will be much simpler if models represent
  real-world entities
• Easier to understand and reuse
• Easier to map to NIEM, which is based on
  Real-World Concepts
• Objects should be normalized and cohesive
• Arbitrary aggregations of information is not ideal

40
Unified Modeling Language (UML)

• UML is an extensive specification for modeling
• Most of it is not relevant to modeling exchange
  document structure
• Focus on static structureclass diagrams

41
Semantic Definitions

• Preservation is key
• In addition to determining a model describing the
  classes and associations, it is critical to
  capture the precise semantic meaning of each
• Class
• Attribute
• Association
• Most UML tools have a means of capturing the
  relevant semantic documentation of each

42
IEPD Process
43
Mapping

• Associates exchange content model concepts to
  NIEM concepts
• Identifies those parts from the exchange model
  that do not readily fit into NIEM
• Missing or incomplete types and elements

44
Mapping Purpose

• Mapping output facilitates the construction of
  exchange schemas
• Defines extensions to NIEM
• Mapping output serves as roadmap for the
  implementer (consumers or providers)
• Explicitly shows where and how business elements
  are represented in the IEP.

45
Mapping to NIEM

• To build schema, each class/property in the
  exchange model must be mapped to a type/element
  in NIEM
• Capture all business rules sometimes
• Mapping can be represented in path-like notation
• It can only be described in prose
• Sometimes exchange concepts are missing from NIEM
• These are mapped to elements in an extension
  schema

46
The Mapping Golden Rule

• Preserve semantic meaning and structure of your
  exchange model
• Force fitting
• Corrupts semantic integrity of Data Model
• Makes it harder for understanding of your
  exchange later on
• Increases chances for corrupting data integrity
  over time
• Must be avoided

47
Mapping Tools

• NIEM exploration tools
• NIEM spreadsheet
• Graphical browser
• SSGT
• Documentation tools
• Spreadsheets work well for documenting mapping
• Not the only way, but in widespread use

48
Basic Example
49
Mapping spreadsheet

• Mapping spreadsheet

50
IEPD Process
51
Making a Schema SubsetThe BIG Picture

• Thus Far
• Created an exchange content model to reflect
  requirements
• Mapped requirements to NIEM elements
• Next
• Create a schema subset that contains just those
  elements

52
Large Data Model

• The National Data Model is large
• Thousands of properties
• Hundreds of types
• ResultLARGE SCHEMA
• Slow validation
• Difficult to manually search
• Difficult to navigate relationships

53
Limited Area of Interest
What I care about
NIEM
54
The Schema Subset Golden Rule

• Instances that validate against the NIEM schema
  subset will validate against the full NIEM schema

55
Subset Purpose?

• Strictly for performance enhancement
• Subsetting should not be viewed as a way for
  enforcing business requirements
• Business requirements require formal documentation

56
SSGT

• Subset Schema Generation Tool
• Exploration tool that creates a valid subset
• http//niem.gtri.gatech.edu/niemtools/ssgt/index.i
  epd

57
Exchange Schemas

• Content is driven by output of mapping step of
  IEPD process
• 3 basic types
• Extension schema
• NIEM Data Model extensions
• Defines some or all of data exchange structure
• Optional, but recommended since it promotes reuse
  of extensions
• Document schema
• Defines some or all of data exchange structure
• Constraint schema
• Validates an exchange using basic XSD validation

58
Extension Schema

• Requires familiarity with how to extend the NIEM
  (XML)
• Defines some or all of data exchange structure
• Optional, but recommended since it promotes reuse
  of extensions

59
Extension Schema Shell (1)
lt?xml version"1.0" encoding"UTF-8"?gt ltxsdschema
targetNamespace"http//mynamespace.com/extension
/1.0" xmlnslocal-ns"http//mynamespace.com/ext
ension/1.0" xmlnsxsd"http//www.w3.org/2001/XMLS
chema" xmlnsnc"http//niem.gov/niem/niem-core/
2.0" xmlnsi"http//niem.gov/niem/appinfo/2.0" e
lementFormDefault"qualified" attributeFormDefault
"unqualified"gt ltxsdannotationgt ltxsddocumenta
tiongtSample extension schemalt/xsddocumentationgt
ltxsdappinfogt ltiConformantIndicatorgttruelt/iC
onformantIndicatorgt lt/xsdappinfogt lt/xsdannota
tiongt
Extension Namespace
Reference required NIEM W3C Namespaces
Required statement of NIEM conformance
60
Extension Schema Shell (2)
Import required NIEM W3C Namespaces
ltxsdimport namespace"http//niem.gov/niem/niem-
core/2.0" schemaLocation"niem/niem-core/2.0/niem-
core.xsd"/gt lt!-- New types to be added
here --gt lt!-- New elements to be added
here--gt lt/xsdschemagt
SchemaLocation either references NIEM subset or
constraint schemas
Reusable types go here
Reusable elements go here
61
Document Schema

• Defines the root element of an exchange
• Brings together all pieces of an exchange

62
Document Schema Shell
lt?xml version"1.0" encoding"UTF-8"?gt ltxsdschema
targetNamespace"http//mynamespace.com/MyExchang
e/1.0" xmlnslocal-ns"http//mynamespace.com/e
xtension/1.0" xmlnsxsd"http//www.w3.org/2001
/XMLSchema" elementFormDefault"qualified"
attributeFormDefault"unqualified"gt ltxsdimport
namespace" http//mynamespace.com/extension/1.0
schemaLocation"ExtensionSchema.xsd"/gt lt!--
New Element Definitions to be added here
--gt ltxselement name"EmergencyResponseRequest"
type"local-ns EmergencyResponseReque
stType"/gt lt/xsschemagt
63
Constraint Schema

• Provides an additional level of constraint,
  beyond the subset schema
• Constraint schemas are optional
• Using them to encapsulate cardinality is usually
  a good idea

64
ConstraintEncapsulating Cardinality

• Allows for cardinality and data format validation
  via XSD
• May simply be a copy of the subset schema with
  minOccurs and maxOccurs attributes set
• When validating, takes the place of the subset
  schema
• Cardinality can also be enforced solely within
  the subset schema, at the cost of easy reuse

65
Schema Overview
66
NIEM Tools

• NIEM Spreadsheet
• NIEM Graphical Browser
• Subset Schema Generation Tool
• NIEM Mapping Tool
• IEPD Tool
• Code List Generator
• Migration Assistance
• NIEM Wayfarer

67
NIEM Spreadsheet

• Microsoft Excel Spreadsheet
• Complete Data Model
• Each Namespace Contained in Tab
• Basic find functionality for searching
• Hyperlinks for navigation
• Download from http//niem.gov (downloads)

68
NIEM Graphical Browser

• Displays small portions of the NIEM data model to
  visualize how things are connected in NIEM.
• Browse properties, types, and associations (the
  nodes) and their their relationships (edges).
• Access begins from any one of a small set of key
  starting points within the model.
• Download from http//niem.gov (tools)

69
Subset Schema Generation Tool

• Provides text-based model navigation
• Navigate through the properties, types, and
  facets and their relationships.
• Provides advanced search configuration options to
  refine search results.
• Creates a schema subset of NIEM components to use
  in an exchange
• Work is maintained in a wantlist
• Download from http//niem.gov (tools)

70
NIEM Mapping Tool

• Allows the user to specify metadata and upload
  XMI domain models associated with NIEM IEPDs.
• Map components within exchange content models to
  NIEM components.
• Generates artifacts based on mappings, including
  mapping reports, wantlist, exchange schemas,
  extension schemas, and subset schemas.
• Access from http//niem.gov (tools)

71
Code List Generator

• Code lists restrict values of XML simple types
• Generates a standard NIEM conformant code list
  schema from an Excel or CSV file.
• A single Excel file can be used to generated
  multiple code types and associated values in one
  schema.
• For separate schemas, use separate Excel files.
• Access from http//niem.gov (tools)

72
Migration Assistance

• Upload and convert GJXDM 3.0.3 to NIEM 2.0
  wantlist and NIEM 1.0 to NIEM 2.0 wantlist
• Outputs NIEM 2.0 wantlist, NIEM 2.0 subset, and
  migration report.
• Migration report contains actions taken and
  choices made in migrating the wantlist, issues
  that cannot be resolved automatically, and
  statistics indicating degree of migration
  resolution.
• Attempts to make as many automatic decisions as
  possible.
• Cannot migrate extension or constraint schemas
• Requires user review for accuracy
• Access from http//niem.gov (tools)

73
NIEM Wayfarer

• Basic GJXDM Wayfarer functionality, only now NIEM
• Detailed Element/Type Information
• Advanced Searching
• Graphical Interface (UML-ish)
• Element Comparison Matrix
• Pseudo-Semantic Searches
• search for cargo and get VehicleCargoText
• search for vessel and get BoatMakeCode
• http//www.ncsconline.org/niemwayfarer

74
NIEM Conformance

• DOJ Special Conditions Language
• Requires conformance with the NIEM
• DHS Special Conditions Language
• Mirrors the DOJ language
• DOJ-DHS Memorandum of Understanding
• Executive Order 13356, replaced by E.O. 13388
  (October 25, 2005)
• Establishes an Information Sharing Council
• Focus on information sharing and interoperable
  systems

75
Flavors of Conformance

• Semantics Conformance
• Technical Conformance
• Vendor/Product Conformance

76
Semantic Conformance

• What do we mean by conformance?
• Conformance means the ability to share accurate
  and reliable information using the NIEM such that
  the information has the same meaning for the
  sender as it does for the receiver

77
Technical Conformance

• Import and reference NIEM namespace or a correct
  subset
• If it exists, use the appropriate NIEM component
  (i.e., do not create a duplicate of one that
  exists)
• Be semantically consistent
• Use NIEM components in accordance with their
  definitions
• Do not use an element to represent data other
  than what its definition describes

78
Technical Conformance (continued)

• Apply XML Schema extension rules correctly and
  consistently
• Naming and Design Rules (NDR)
• Exchanges conform, NOT systems
• What you call data or how you use it in your own
  system does not impact conformance
• What counts is how you package data as XML for
  exchange
• There is no concept of partial conformance

79
Conformant Products?

• Remember Exchanges conform to NIEM, products,
  applications, etc. are not NIEM-conformant
• A product, application, etc. should instead have
  the ability to produce and/or consume XML that
  conforms to an IEPD

80
Questions?
81
Contact

• Andrew Owen
• Justice Information Systems Specialist
• SEARCH
• Andrew_at_search.org