Master NIEM Training Document

NIEM Training

Tom Carlson, GTRI Thomas.Carlson@gtri.gatech.edu

Introduction

Purpose

A technical webinar focused on developers and implementers who are interested in a deep-dive of the National Information Exchange Model.

Supporting Documents

All materials are available on the NIEM Training Github repo at https://github.com/NIEM/NIEM-Training. Specific materials used are listed on the README page and include:

Master NIEM Training Document
Mapping Spreadsheets
Ersatz Textual Instances

Sample Message Specification

Crash Driver Message Specification

Agenda

Logistics
Introduction to NIEM
Introduction to Message Spec / IEPD Development
- Scenario Planning
- Requirements Analysis
- Mapping
  - Intro to Mapping
  - Mapping to Existing Objects
  - Creating New Objects
  - External Standards
- Creating and Validating Schemas
- Assembly
- Publishing
- Implementation
Exercises
Resources

Logistics and Background

Major revamp of prior training programs
Three days, 1-5pm each day
Short breaks on the hour
Ask questions via chat when you have them
- Someone is monitoring chat and will interrupt me as needed
This document and supporting materials are at:
- https://github.com/niem/niem-training

Introduction to NIEM

What is NIEM?
The Scope of NIEM
NIEM Harmonization and Organization

What is NIEM? - Framework

NIEM is a community-driven, government and jurisdiction-wide, standards-based approach to exchanging information
Diverse communities can collectively leverage NIEM to increase efficiencies and improve decision-making
NIEM is available to everyone, including public and private organizations
NIEM includes a data model, governance, training, tools, technical support services, and an active community to assist users in adopting a standards-based approach to exchanging data

What is NIEM? – Interop Problem

If we don’t understand what each other means, we won’t be able to exchange info
Need a common language for defining things

What is NIEM? And what not?

NIEM is:

a common vocabulary
a means of enabling efficient information exchange across diverse public and private organizations

NIEM is not:

a system or database
a means of specifying how to transmit or store data

What is NIEM? – Exchanging Data and Components

Using NIEM, organizations come together to agree on a common vocabulary
When additional organizations are added to the information exchange, the initial NIEM exchange can be reused, saving time and money

What is NIEM? – Simplified Exchanges

When using NIEM, you only need to “speak” two languages: your own and NIEM

Without NIEM	With NIEM

Scope of NIEM

NIEM is a data layer standard and intentionally does not address all the necessary technologies needed for information sharing
Exchange partners decide how to store and process the NIEM-conformant data being exchanged

Scope of NIEM - Open Source Interconnection (OSI) model

Detailed:

Simplified:

Bottle of Liquid

A bottle of liquid:


Has some combination of liquids inside	NIEM is the liquid inside, the payload document
Has a shape	NIEM doesn’t care about the shape
Made of certain materials	NIEM doesn’t care about what the bottle is made out of or how it's constructed
Can be opaque or transparent	NIEM doesn’t care about whether you can see into the bottle
Is moved around by various means	NIEM doesn’t care about how you move the bottle around
Can be filled and emptied	NIEM doesn’t care about how you filled it or what you do with the liquid later

NIEM Harmonization and Organization

Think of the NIEM data model as a mature and stable data dictionary of agreed-upon terms, definitions, relationships, and formats independent of how information is stored in individual agency systems
The data model consists of two sets of closely related vocabularies:
- NIEM core
- Individual NIEM domains
NIEM core includes data elements commonly agreed upon across all NIEM domains (i.e., person, activity, location, and item, etc.)
Individual NIEM domains contain mission-specific data components that build upon NIEM core concepts

Existing Domains	Upcoming Domains
Agriculture	Learning and Development
Biometrics	International Human Services
Chemical, Biological, Radiological, & Nuclear
Cyber (new in 5.1)
Emergency Management
Human Services
Immigration
Infrastructure Protection
Intelligence
International Trade
Justice
Maritime
Military Operations
Screening
Surface Transportation

Domains hold objects specific to their domains:

Agriculture	Biometrics

Chemical, Biological, Radiological, & Nuclear	Emergency Management

Human Services	Immigration

Infrastructure Protection	Intelligence

International Trade	Justice

Maritime	Military Operations

Screening	Surface Transportation

NIEM Versioning

NIEM has major and minor versions, plus domain updates
Major version releases, e.g. 4.2 to 5.0
- Every 3 years
- All bets are off
- NIEM-Core can and will change
  - Underlying infrastructure can also change
- Domains can change
- Domains are harmonized
  - Combination of tools and human collaboration to reach consensus
  - Repeated content is collapsed
  - Misplaced content is moved, either to other domains or to core
  - New content is added
- Nothing in a major version change is guaranteed to be backwards compatible with earlier major releases
- NIEM 6.0 scheduled for end of 2023
Minor version releases, e.g. 5.0 to 5.1:
- Annually
- NIEM-Core does not change!
  - Neither does the underlying infrastructure
- Domains can change
- Domains can be harmonized
- Domains can be added, like Cyber in 5.1
- Domains are not guaranteed to be backwards compatible with earlier minor releases
  - But they often are
- NIEM 5.2 scheduled for end of 2022
Domain updates are done per-domain
- Domains can update their content in between minor releases
- Those updates then are normally folded into the next minor release
Older versions never go away
- You can still use NIEM 1.0 (but shouldn't)
- There are plenty of NIEM 2.0 exchanges in current use
Migration
- Don't have to migrate
- May want to migrate if a newer version gives you functionality you need (and you're already making changes)
- NIEM provides tools for migrating the NIEM objects, ~90% effective
- Manual work is needed for things you've added for your exchange

NIEM Administration and Organization

Move to OASIS

The NIEM program is transitioning from being a solely government-funded project to an open source project under OASIS Open. So here's a quick plug for OASIS Open...

About OASIS Open

One of the most respected, nonprofit open source and open standards bodies in the world, OASIS Open advances the fair, transparent development of open source software and standards through the power of global collaboration and community.

Why Should My Organization Join the NIEM OASIS Open Project?

Participate in the development of internationally recognized NIEM standards, ensuring your perspectives and use-cases are represented
Unify fragmented efforts and encourage convergence
Facilitate communication between private and public-sector
Create partnerships and tap into brain trust of NIEM experts from the public sector
Opportunity to become a steward with a seat on the NIEM Board
Determine if/when approved work should be submitted to ITU, ICC, UN, ISO, etc.

Get Involved!

For more information about supporting the NIEM OASIS Open Project, contact Dee Schur, dee.schur@oasis-open.org

Message Spec / IEPD Overview

"Information Exchange Package Documentation"
Slowly changing to "Message Specification"
Defines an exchange
Made up of a bunch of documents, "artifacts"
- Some meant for humans
- Some meant for computers

Message Spec Process

Creating a Message Spec is a multi-step process:

Scenario Planning
Analyze Requirements
Map and Model
Build and Validate
Assemble and Document (and Publish)
Implement

Each step produces artifacts used by subsequent steps:

A clue as to what you're doing...
UML Diagrams
Mapping Spreadsheet
Schemas and Instance Documents
Textual Documents
Code

Message Spec / IEPD Process – Idealized

Message Spec / IEPD Process – Real Life

Message Spec / IEPD Artifacts - Documentation

Master Documentation (Word)
Message Spec / IEPD catalog document (iepd-catalog.xml)
Change log (text)
README (text)
Conformance assertion (text)
Mapping Spreadsheet

Message Spec / IEPD Artifacts - Definitional

Wantlist (wantlist.xml)
Schema subset schemas
Extension schemas
Exchange schemas
Sample instances
XML catalogs

Scenario Planning

Decide what the exchange is about
Who are the exchange partners?
Who are stakeholders?
Communication is key

sequenceDiagram
	Sender ->> Receiver: So we send you A, B, and C.
	Receiver ->> Sender: We don't use C. We throw it away. We really need D.
	Sender ->> Receiver: What? C is hard to generate. D is easy!
	Receiver ->> Sender: ¯\_(ツ)_/¯

Loading

Existing exchanges or other documentation can help
- Within your organization
- From NIEM repositories (huge caveat)
- There's a community to draw from

Existing Documentation

Current technical architecture documents of all exchange partners
Stakeholders that will be involved in the exchange
Security, privacy, and other policy-related concerns associated with the exchange
Technical characteristics of the exchange:
- Types of data being shared
- Number of data objects
  - Current structure of the data (logical, physical)
  - Use of external standards

Analyze Requirements

Diagrams
- Use Case Diagrams
- Business Process Diagrams
- Sequence Diagrams
Class Diagrams
Spreadsheets
Other documents

Use Case Diagrams

Business Process Diagrams

Sequence Diagrams

UML Class Diagrams

The bread and butter of Message Spec / IEPD business requirements
Can be oriented towards business terms and objects
- Better for consensus
Can be oriented towards NIEM terms and objects
- Pre-loads the mapping step

Business Oriented Class Diagram

Representing objects in UML by their business names and relationships.

This vary depending on how your subject matter experts view objects and their relationships. For example, the charge to person relationship could be charge to the driver instead. Cardinality, which we will touch on later, could vary depending on the needs of the exchange. The important thing is reflecting how your business folks understand the data, not model purity.

NIEM Oriented Class Diagram

UML Tools

ArgoUML
BOUML
MagicDraw / Rational Rose ($$$)
Visio / OmniGraffle
Graphviz / Mermaid / PlantUML
Many more…

Business Rules

Some rules are not easily represented in UML
You can use whatever works for your needs
Schematron for XML
- e.g. Birth dates must be in the past, salutations matching gender
Plain old textual descriptions are great!

The Process from Here On…

We have choices on how to proceed:

Step-by-step
- Finish mapping entirely before starting schemas
Concurrent
- Building schemas and instances as you go
We'll use something in-between
- Build an ersatz matching instance document as we map (Crash Driver Report Complete)
- Sorta like YAML without data values
- Save schemas for the end

(Will use a super secret tool to help with some of this!)

XML Schema in a Nutshell

XML Schema defines what an XML document needs to look like. (JSON Schema does the same for JSON documents.)

For example, this bit of XML Schema defines what a PersonName object needs to look like.

<xs:complexType name="PersonNameType">
	<xs:annotation>
		<xs:documentation>A data type for a combination of names and/or titles by which a person is known.</xs:documentation>
	</xs:annotation>
	<xs:sequence>
		<xs:element name="PersonGivenName" type="xs:string" minOccurs="1" maxOccurs="1"/>
		<xs:element name="PersonMiddleName" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
		<xs:element name="PersonSurName" type="xs:string" minOccurs="1" maxOccurs="1"/>
	</xs:sequence>
	<xs:attribute name="personNameCommentText" type="xs:string" use="optional"/>
</xs:complexType>

<xs:element name="PersonName" type="PersonNameType" nillable="true">
	<xs:annotation>
		<xs:documentation>A combination of names and/or titles by which a person is known.</xs:documentation>
	</xs:annotation>
</xs:element>

And here's what the matching XML instance document might look like.

	<PersonName personNameCommentText="copied">
		<PersonGivenName>Peter</PersonGivenName>
		<PersonMiddleName>Bredon</PersonMiddleName>
		<PersonMiddleName>Death</PersonMiddleName>
		<PersonSurName>Wimsey</PersonSurName>
	</PersonName>

We will see much more of this. This is just an initial glance at it.

Map and Model

For this entire section, we'll look at various things in the mapping spreadsheet and show how to map them to NIEM or to new elements that we'll create later. As we move through, we'll cover all the major aspects of how NIEM works.

Introduction to Mapping

Mapping Spreadsheets Options
- NIEM Mapping Template
- Simple Training Spreadsheet
- Something In-between
Document Business Objects
Map them to NIEM objects, existing or new
Maintaining an Ongoing Sample Instance Skeleton

NIEM Mapping Template

Primarily for submitting content for inclusion in NIEM
- Eight different tabs
Can also be used for mapping in a Message Spec / IEPD
- Just need a few of the tabs, mainly Property and Type
Is a bit overkill for a Message Spec
Slight difference between the versions on the NIEM site and for use with MEP Builder
Mapping Spreadseet Template

Simple Training Spreadsheet

Minimal
Designed to be simple enough to fit on slides
- No, really, that’s the constraint
Usually expanded in practice

Tom’s Custom Mapping Spreadsheet

Has evolved over time
Contains all the info needed to make schemas
Not as overwhelming as the NIEM Mapping Template
Works with the NIEM Linter
You can make your own custom one
- The IEPD Spec doesn’t specify a required format, by design
Fresh copy for our example Message Spec / IEPD
Check on the cardinality
- Why is Person 1..1?
- Because this is a report on a Crash Driver, not the Crash itself
  - So the driver is the only person
  - Could simplify this Message Spec with some assumptions
  - But that would leave us with too little to do in class
- A different context would have different cardinality
  - Likely multiple Person objects

Basics of Searching NIEM

Tools

SSGT
Wayfarer
NIEM Schemas
- Official Releases
- HyperNIEM
Spreadsheet (included in the official releases)

Techniques

Search for Terms
- Simple vs Advanced
Search for Synonyms
Search for Word Roots
Search for Containers
Search for Properties

Mapping and NIEM Technical Details

Understanding NIEM Objects

XML Schema uses elements and types
What an element can hold is based on its type
What you are (your type) defines what you can hold

XML Schema

The XML Schema defining nc:Person includes a definition and a type.

Its type, nc:PersonType, has a little more information. It also includes a definition, one similar to nc:Person. It also includes a base that tells us what sort of thing it is. In this case, the base is structures:ObjectType, which is just an empty object (that has a few infrastructure pieces we'll learn about later). To that base it adds several objects. These are objects that go inside an nc:Person object. Each one is a reference to a declaration of each of those objects. Each also has cardinality defined, which tells us how many of each can go inside of an nc:Person. minOccurs is the minimum number of times. Any non-negative integer can go here, but 0 and 1 are what you'll usually find. Zero essentially means "optional." maxOccurs is the maximum number of times. This can also be any non-negative number, but can also be "unbounded", which means "as many as you want." Typical values are 1 and unbounded. Here's the schema for nc:PersonType with some of the contained objects removed for clarity:

<xs:complexType name="PersonType">
	<xs:annotation>
		<xs:documentation>A data type for a human being.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:ObjectType">
			<xs:sequence>
				<xs:element ref="nc:PersonAccentText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonAgeDescriptionText" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="nc:PersonLivingIndicator" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="nc:PersonName" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="nc:PersonHomeContactInformation" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

<xs:element name="Person" type="nc:PersonType" nillable="true">
	<xs:annotation>
		<xs:documentation>A human being.</xs:documentation>
	</xs:annotation>
</xs:element>

The XML Schema defining nc:PersonName and nc:PersonNameType looks like:

<xs:complexType name="PersonNameType">
	<xs:annotation>
		<xs:documentation>A data type for a combination of names and/or titles by which a person is known.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:ObjectType">
			<xs:sequence>
				<xs:element ref="nc:PersonNamePrefixAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonGivenName" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonMiddleName" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonSurName" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonNameSuffixText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonMaidenName" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonFullName" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonNameCategoryAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonNameSalutationText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonOfficialGivenName" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonPreferredName" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonSurNamePrefixText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:EffectiveDate" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:PersonNameAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
			<xs:attribute ref="nc:personNameCommentText" use="optional"/>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

<xs:element name="PersonName" type="nc:PersonNameType" nillable="true">
	<xs:annotation>
		<xs:documentation>A combination of names and/or titles by which a person is known.</xs:documentation>
	</xs:annotation>
</xs:element>

The XML Schema defining nc:PersonGivenName, PersonNameTextType, and supporting types looks like:

<xs:complexType name="PersonNameTextType">
	<xs:annotation>
		<xs:documentation>A data type for a name by which a person is known, referred, or addressed.</xs:documentation>
	</xs:annotation>
	<xs:simpleContent>
		<xs:extension base="nc:ProperNameTextType">
			<xs:attribute ref="nc:personNameInitialIndicator" use="optional"/>
		</xs:extension>
	</xs:simpleContent>
</xs:complexType>

<xs:complexType name="ProperNameTextType">
	<xs:annotation>
		<xs:documentation>A data type for a word or phrase by which a person or thing is known, referred, or addressed.</xs:documentation>
	</xs:annotation>
	<xs:simpleContent>
		<xs:extension base="nc:TextType"/>
	</xs:simpleContent>
</xs:complexType>

<xs:complexType name="TextType">
	<xs:annotation>
		<xs:documentation>A data type for a character string.</xs:documentation>
	</xs:annotation>
	<xs:simpleContent>
		<xs:extension base="niem-xs:string">
			<xs:attribute ref="nc:partialIndicator" use="optional"/>
			<xs:attribute ref="nc:truncationIndicator" use="optional"/>
			<xs:attribute ref="xml:lang" use="optional"/>
		</xs:extension>
	</xs:simpleContent>
</xs:complexType>

<xs:element name="PersonGivenName" type="nc:PersonNameTextType" nillable="true">
	<xs:annotation>
		<xs:documentation>A first name of a person.</xs:documentation>
	</xs:annotation>
</xs:element>

Finally, the definition for nc:PersonLivingIndicator is very simple. It's a boolean value, true or false.

<xs:element name="PersonLivingIndicator" type="niem-xs:boolean" nillable="true">
	<xs:annotation>
		<xs:documentation>True if a person is alive; false if a person is dead.</xs:documentation>
	</xs:annotation>
</xs:element>

The resulting instance document that all this creates might look like this:

<nc:Person>
	<nc:PersonLivingIndicator>false</nc:PersonLivingIndicator>
	<nc:PersonName nc:personNameCommentText="copied">
		<nc:PersonGivenName>Peter</nc:PersonGivenName>
		<nc:PersonMiddleName>Death</nc:PersonMiddleName>
		<nc:PersonMiddleName>Bredon</nc:PersonMiddleName>
		<nc:PersonSurName>Wimsey</nc:PersonSurName>
	</nc:PersonName>
</nc:Person>

(Who is Peter Wimsey?)

What About JSON?

NIEM can be used with JSON via JSON-LD. JSON-LD extends JSON to allow for meaningfully linking data together. NIEM leverages JSON-LD in two ways:

NIEM uses @context to provide a mapping from JSON object names back to their counterparts in NIEM
NIEM uses @id to provide links between JSON objects

Linking is a topic for later in the class, but here is an example of the @context. This context tells us that nc:Person refers to the Person object in the NIEM-Core namespace.

{
	"@context": {
		"nc": "http://release.niem.gov/niem/niem-core/5.0/#"
	},
	"nc:Person": {
		"nc:PersonLivingIndicator": "false",
		"nc:PersonName": {
			"nc:personNameCommentText": "copied",
			"nc:PersonGivenName": "Peter",
			"nc:PersonSurName": "Wimsey"
		}
	}
}

There are other means to shorten long NIEM names into names more amenable to JSON developers via normalization, but that topic is outside the scope of the training.

Note that the structure of the JSON mirrors the structure of the XML. NIEM with JSON does require mirroring the structure.

JSON-LD and JSON Schema are separate concepts. NIEM does not yet support JSON Schema, although efforts are underway to enable that. Currently, JSON with NIEM is all about the JSON instance documents. NIEM itself is still defined in XML Schema.

Throughout the training, matching JSON instances will be included with XML instances.

Native Properties

Some things will be easy to find
Will map directly to NIEM objects
Examples:
- nc:Person, nc:PersonName, nc:PersonGivenName, etc.
- Search for nc:Person in the SSGT
- Search for nc:Person in Wayfarer

We've already seen the schema for these in detail.

Artifacts

Native Properties
Mapping Spreadsheets

Substitution Groups

Some concepts can be represented multiple ways
Text / code combinations are common
Date can be a date, datetime, or a range
NIEM uses substitution groups to support both:
- The single concept, and
- Multiple representations of that concept
Examples:
- nc:PersonBirthDate (SSGT/Wayfarer) contains a nc:DateRepresentation (SSGT/Wayfarer)
- Substitution group heads follow the form of: SomethingRepresentation or WhateverAbstract

Schemas

Here we see nc:PersonBirthDate and its type, nc:DateType:

<xs:element name="PersonBirthDate" type="nc:DateType" nillable="true">
	<xs:annotation>
		<xs:documentation>A date a person was born.</xs:documentation>
	</xs:annotation>
</xs:element>

nc:DateType contains nc:DateRepresentation, along with other date properties:

<xs:complexType name="DateType">
	<xs:annotation>
		<xs:documentation>A data type for a calendar date.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:ObjectType">
			<xs:sequence>
				<xs:element ref="nc:DateRepresentation" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:DateAccuracyAbstract" minOccurs="0" maxOccurs="1"/>
				<xs:element ref="nc:DateMarginOfErrorDuration" minOccurs="0" maxOccurs="1"/>
				<xs:element ref="nc:DateAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

nc:DateRepresentation is abstract, meaning it has no type and must be substituted with another object:

<xs:element name="DateRepresentation" abstract="true">
	<xs:annotation>
		<xs:documentation>A data concept for a representation of a date.</xs:documentation>
	</xs:annotation>
</xs:element>

Those other objects are identified by having a substitutionGroup attribute set to the name of the substitution group head. The most common one you'll see for nc:DateRepresentation is nc:Date:

<xs:element name="Date" type="niem-xs:date" substitutionGroup="nc:DateRepresentation" nillable="true">
	<xs:annotation>
		<xs:documentation>A full date.</xs:documentation>
	</xs:annotation>
</xs:element>

Instance Documents

In the resulting instance document, you don't see nc:DateRepresentation at all. You just see nc:Date, which is taking its place:

<nc:Person>
	<nc:PersonBirthDate>
		<nc:Date>1890-05-04</nc:Date>
	</nc:PersonBirthDate>
	<nc:PersonName>
		<nc:PersonGivenName>Peter</nc:PersonGivenName>
		<nc:PersonMiddleName>Death</nc:PersonMiddleName>
		<nc:PersonMiddleName>Bredon</nc:PersonMiddleName>
		<nc:PersonSurName>Wimsey</nc:PersonSurName>
	</nc:PersonName>
</nc:Person>

The matching JSON similarly just shows nc:Date:

"nc:Person": {
	"nc:PersonBirthDate": {
		"nc:Date": "1890-05-04"
	},
	"nc:PersonName": {
		"nc:PersonGivenName": "Peter",
		"nc:PersonMiddleName": [
			"Death",
			"Bredon"
		],
		"nc:PersonSurName": "Wimsey"
	}
}

Artifacts

Substitition Groups
Mapping Spreadsheets

An aside about Namespaces

Below you can see object with different prefixes, j:Crash and nc:ActivityDate. The j and nc refer to different namespaces in XML Schema.

Namespaces organize elements by context
Identified by prefix, a nickname for the namespace
NIEM governance is organized along these lines

<xs:schema targetNamespace="http://training.niem.gov/CrashDriver/1.0/extension" version="1" xml:lang="en-US"  
    xmlns:ext="http://training.niem.gov/CrashDriver/1.0/extension"  
    xmlns:j="http://release.niem.gov/niem/domains/jxdm/7.0/"  
    xmlns:nc="http://release.niem.gov/niem/niem-core/5.0/">

    <xs:import schemaLocation="../niem/domains/jxdm.xsd"  
        namespace="http://release.niem.gov/niem/domains/jxdm/7.0/"/>
	<xs:import schemaLocation="../niem/niem-core.xsd"  
        namespace="http://release.niem.gov/niem/niem-core/5.0/"/>

JSON-LD maps JSON objects to NIEM namespaces in the @context. Each of these entries maps a prefix to a NIEM namespace, providing a link back to the NIEM object.

"@context": {
	"ext": "http://training.niem.gov/CrashDriver/1.0/extension#",
	"j": "http://release.niem.gov/niem/domains/jxdm/7.0/#"
	"nc": "http://release.niem.gov/niem/niem-core/5.0/#",
}

As mentioned earlier, NIEM doesn't support JSON Schema well yet. Using NIEM with JSON is currently focused on creating matching instance documents. Upcoming NIEM developments will greatly enhance the ability to work in JSON as a similar level as with XML and XML Schema.

Inherited Properties

NIEM is a model, not a flat data dictionary
Some concepts don’t exist as elements using the terms for the concept
Instead, properties are inherited
There is no “Crash Date” in NIEM (SSGT/Wayfarer)
There is an ActivityDate which can be inside a Crash object (SSGT/Wayfarer)

Schemas

Here's j:Crash and its type, j:CrashType:

<xs:element name="Crash" type="j:CrashType" nillable="true">
	<xs:annotation>
		<xs:documentation>A traffic accident.</xs:documentation>
	</xs:annotation>
</xs:element>

j:CrashType contains several things, but the important thing here is what it's based on, j:DrivingIncidentType:

<xs:complexType name="CrashType">
	<xs:annotation>
		<xs:documentation>A data type for a traffic accident.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="j:DrivingIncidentType">
			<xs:sequence>
				<xs:element ref="j:CrashServiceCall" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:CrashInformationSource" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="j:CrashAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

j:DrivingIncidentType is, in turn, based on an even more generic type, nc:IncidentType:

<xs:complexType name="DrivingIncidentType">
	<xs:annotation>
		<xs:documentation>A data type for details of an incident involving a vehicle.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="nc:IncidentType">
			<xs:sequence>
				<xs:element ref="j:DrivingAccidentSeverityAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DrivingIncidentCMVAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="j:DrivingIncidentAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

nc:IncidentType is, also in turn, based on a very generic type, nc:ActivityType:

<xs:complexType name="IncidentType">
	<xs:annotation>
		<xs:documentation>A data type for an occurrence or an event that may require a response.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="nc:ActivityType">
			<xs:sequence>
				<xs:element ref="nc:IncidentEvent" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:IncidentJurisdictionalOrganization" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="nc:IncidentAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

And we finally get to nc:ActivityType, which contains nc:ActivityDate:

<xs:complexType name="ActivityType">
	<xs:annotation>
		<xs:documentation>A data type for a single or set of related actions, events, or process steps.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:ObjectType">
			<xs:sequence>
				<xs:element ref="nc:ActivityIdentification" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:ActivityActualDuration" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="nc:ActivityDate" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="nc:ActivityAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

What this all means is that a j:Crash object can contain a nc:ActivityDate, which is, contextually, a Crash Date.

Instance Documents

Instance in XML:

<j:Crash>
	<nc:ActivityDate>
		<nc:Date>1900-05-04</nc:Date>
	</nc:ActivityDate>
</j:Crash>

Instance in JSON:

"j:Crash": {
	"nc:ActivityDate": {
		"nc:Date": "1900-05-04"
	}
}

You need to understand this concept in order to know to look for these cases, which are very common, but just use tools to figure out the details, e.g:

j:Crash in the SSGT
j:Crash in Wayfarer
- Wayfarer has a new feature that does some level of contextual searching
- Search contextually for "crash date" in Wayfarer

Artifacts

Inherited Properties
Mapping Spreadsheets

Linking Things Together

NIEM is relational in many senses
Objects can refer to each other across an XML hierarchy
Allows NIEM to represent real world, many-to-many relationships
structures namespace provides the infrastructure to make this work

NIEM Structural "Layers"

NIEM has several conceptual layers which build on top of each other:

structures provides infrastructure
- All namespaces draw from it
niem-core provides common objects to be used or built upon
- Domains draw from it
Domains provide domain-specific content, often built from niem-core
- Domains can draw from each other, although this is limited in practice
Code table namespaces define many of the code tables, built with infrastructure from structures
- Domains draw from it for code definitions
Wrappers for external standards, built with infrastructure from structures

Referencing - XML Schema

NIEM lets you assign unique IDs to objects
Other objects can then link to those objects by referencing the ID
Everything in NIEM can have attributes for this:
- id
- ref
- metadata
These leverage built-in XML Schema attributes ID, IDREF, and IDREFS
id assigns an ID to an object
- Just a string
- Must be unique within an instance document
- Can't include a space
ref references an ID of another object
- Contains a single id to match
- The matching id must exist in the instance document
- Validators do not check that the linking makes sense
metadata references IDs of metadata object
- Conceptually, an object is saying that this is the metadata that applies to itself
- Can contain multiple IDs, separated with spaces
- The matching ids must exist in the instance document

Examples of how NIEM uses these are the next few sections. Here's a very simple non-NIEM example:

Plain Old XML:

<Cat>
	<CatName>Jett</CatName>
	<CatOwner IDREF="owner01"/>
</Cat>
<Human ID="owner01">
	<HumanName>Tom</HumanName>
</Human>

NIEM's renaming:

<Cat>
	<CatName>Jett</CatName>
	<CatOwner ref="owner01"/>
</Cat>
<Human id="owner01">
	<HumanName>Tom</HumanName>
</Human>

Referencing - JSON-LD

JSON itself doesn't provide a means of linking objects together. NIEM leverages JSON-LD to provide that functionality. We've seen earlier how the @context section provides a mapping from a JSON instance back to NIEM object. NIEM also uses @id objects to both mark objects with an ID as well as refer to IDs applied to other objects.

Again, we will see example throughout the next few sections.

"Cat": {
	"CatName": "Jett",	
	"CatOwner": {
		"@id": "#owner01"
	}
},
"Human": {
	"@id": "#owner01",
	"HumanName": "Tom"
}

Associations

Relationships can be complex
NIEM provides powerful Association objects
Trade-off can be implementation complexity
Associations link objects together
- Associations reference the associated objects
- Can link to multiple objects
- Can thus form many-to-many relationships between them
Associations also hold information about the association itself
- Date or DateRange is a common example
Associations can also just include them, if applicable
Examples:
- j:PersonChargeAssociation (SSGT/Wayfarer)
- Anything ending in “Association” (SSGT/Wayfarer)

Schemas

j:PersonChargeAssociation is used, as the name suggests, to link together a Person and a Charge. It's of j:PersonChargeAssociationType:

<xs:element name="PersonChargeAssociation" type="j:PersonChargeAssociationType" nillable="true">
	<xs:annotation>
		<xs:documentation>An association between a person and a charge issued to that person.</xs:documentation>
	</xs:annotation>
</xs:element>

j:PersonChargeAssociationType includes a nc:Person and a j:Charge, but also includes information about the association itself, in this case j:JuvenileAsAdultIndicator. (Which isn't used in this exchange.) PersonChargeAssociationType extends nc:AssociationType:

<xs:complexType name="PersonChargeAssociationType">
	<xs:annotation>
		<xs:documentation>A data type for an association between a person and a charge.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="nc:AssociationType">
			<xs:sequence>
				<xs:element ref="nc:Person" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:Charge" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:JuvenileAsAdultIndicator" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:PersonChargeAssociationAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

nc:AssociationType adds in generic association information like a start and end date and a description. Remember that these are inherited by any type based on nc:AssociationType, so things of j:PersonChargeAssociationType automatically get these common objects.

<xs:complexType name="AssociationType">
	<xs:annotation>
		<xs:documentation>A data type for an association, connection, relationship,
			or involvement somehow linking people, things, and/or activities together.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:AssociationType">
			<xs:sequence>
				<xs:element ref="nc:AssociationDateRange" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:AssociationDescriptionText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:AssociationAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

Instance Documents

In the instance document, the association object can specify the objects being associated together by pointing to then with ref attributes, or by including the object inside the association object.

First, here's the association using referencing. The applicable nc:Person and j:Charge objects are external to j:PersonChargeAssociation. While shown next to the j:PersonChargeAssociation object, they could be anywhere in the instance document.

This method is useful when the same nc:Person object is being used in multiple contexts. In the example Message Spec / IEPD, this nc:Person is also the j:CrashDriver and the j:CrashPerson. This method allows us to define the nc:Person once and refer to it multiple times.

This beings with it two advantages:

This nc:Person object is not being replicated; there is no duplication of data
We know that the nc:Person in the Association and the j:CrashDriver and the j:CrashPerson are the exact same person, and not three people with the same name

<j:PersonChargeAssociation>
	<nc:Person structures:ref="P01" xsi:nil="true"/>
	<j:Charge structures:ref="CH01" xsi:nil="true"/>
</j:PersonChargeAssociation>
<nc:Person structures:id="P01">
	<nc:PersonName>
		<nc:PersonGivenName>Peter</nc:PersonGivenName>
	</nc:PersonName>
</nc:Person>
<j:Charge structures:id="CH01">
	<j:ChargeDescriptionText>Furious Driving</j:ChargeDescriptionText>
	<j:ChargeFelonyIndicator>false</j:ChargeFelonyIndicator>
</j:Charge>

The disadvantage with using referencing is that it can be more difficult to implement. Implementations need to look for ref attributes and find matching id attributes.

The alternative to referencing is to just include the nc:Person and j:Charge information inside the association:

<j:PersonChargeAssociation>
	<nc:Person>
		<nc:PersonName>
			<nc:PersonGivenName>Peter</nc:PersonGivenName>
		</nc:PersonName>
	</nc:Person>
	<j:Charge>
		<j:ChargeDescriptionText>Furious Driving</j:ChargeDescriptionText>
		<j:ChargeFelonyIndicator>false</j:ChargeFelonyIndicator>
	</j:Charge>
</j:PersonChargeAssociation>

You can also mix and match. You could reference a nc:Person while including the j:Charge inside the association.

NIEM never requires you to do referencing. If the trade-offs of duplicated data aren't an issue, you can have multiple identical nc:Person objects.

JSON-LD works similarly. j:PersonChargeAssociation contains a nc:Person object with an @id. j:Charge is the same. These IDs match the @id objects in the actual nc:Person and j:Charge object outside the association:

"j:PersonChargeAssociation": {
	"nc:Person": {
		"@id": "#P01"
	},
	"j:Charge": {
		"@id": "#CH01"
	},
},
"nc:Person": {
	"@id": "#P01",
	"nc:PersonName": {
		"nc:PersonGivenName": "Peter"
	}
},
"j:Charge": {
	"@id": "#CH01",
	"j:ChargeDescriptionText": "Furious Driving",
	"j:ChargeFelonyIndicator": false
}

As with the XML version, you can also just include the nc:Person and j:Charge objects inside the association:

"j:PersonChargeAssociation": {
	"nc:Person": {
		"nc:PersonName": {
			"nc:PersonGivenName": "Peter"
		}
	},
	"j:Charge": {
		"j:ChargeDescriptionText": "Furious Driving",
		"j:ChargeFelonyIndicator": false
	}
}

The same trade-offs apply as in XML, but JSON developers may lean more towards inclusion rather than referencing based on implementation effort required. Again, NIEM never requires you to do referencing.

Additionally, while referencing is part of XML Schema, it is not part of JSON Schema. Checking that the @id values match up needs to be done with a JSON-LD validator.

Associations
Mapping Spreadsheets

Roles

Modeling some objects as specialized things gets complicated
Roles are, well, roles that objects play
- People, organizations, items are the major ones
Roles reference the objects playing the role
- An objects can thus play multiple roles, because multiple role objects can point to it
Roles can also just include the object playing the role, as with associations
Roles contain other information about the role
Examples:
- j:CrashPerson (SSGT/Wayfarer)
- Anything containing an nc:RoleOfPerson(SSGT/Wayfarer), nc:RoleOfOrganization (SSGT/Wayfarer), or nc:RoleOfItem (SSGT/Wayfarer), plus a few lesser used ones (SSGT/Wayfarer)

Searching for Injured Person

How do we find the match for "injured person"? Here's one way:

Search for injured person
- SSGT
- Wayfarer
No results! How about person injury?
- SSGT
- Wayfarer
Notice that j:InjuryLocationCode looks like something an injured person would have
That's inside of things of nc:InjuryType
j:CrashPersonInjury is one of the things of that type
And that's inside of j:CrashPerson

Roles - Not Special Kinds of People

Roles	Just a Guy

Roles – Allows an Object to Play Multiple Roles

Roles - How They Work

Roles Point to the Object Playing the Role:

Roles Contain Info About the Role

Schemas

j:CrashPerson is of j:CrashPersonType:

<xs:element name="CrashPerson" type="j:CrashPersonType" nillable="true">
	<xs:annotation>
		<xs:documentation>A person involved in a traffic accident.</xs:documentation>
	</xs:annotation>
</xs:element>

j:CrashPersonType contains a nc:RoleOfPerson object, along with information specific to this role. In the sample Message Spec / IEPD, we're also using j:CrashPersonInjury:

<xs:complexType name="CrashPersonType">
	<xs:annotation>
		<xs:documentation>A data type for any person involved in a traffic accident.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:ObjectType">
			<xs:sequence>
				<xs:element ref="nc:RoleOfPerson" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:CrashPersonEMSTransportation" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:CrashPersonInjury" minOccurs="0" maxOccurs="unbounded"/>
				<!-- A whole slew of objects removed for clarity -->
				<xs:element ref="j:CrashPersonAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

nc:RoleOfPerson is of nc:PersonType. That means it can hold person-specific information, but like everything in NIEM, it can also point at other objects:

<xs:element name="RoleOfPerson" type="nc:PersonType" substitutionGroup="nc:RoleOfAbstract" nillable="true">
	<xs:annotation>
		<xs:documentation>A person of whom the role object is a function.</xs:documentation>
	</xs:annotation>
</xs:element>

Instance Documents

As with associations, we can implement the role in two different ways. Here we see it used as a reference. j:CrashDriver contains a nc:RoleOfPerson, which uses the ref attribute to point to the nc:Person object playing that role:

<j:CrashPerson>
	<nc:RoleOfPerson structures:ref="P01" xsi:nil="true"/>
	<j:CrashPersonInjury>  
		<nc:InjuryDescriptionText>Broken Arm</nc:InjuryDescriptionText>  
	</j:CrashPersonInjury>
</j:CrashPerson>

<nc:Person structures:id="P01">
	<nc:PersonName>
		<nc:PersonGivenName>Peter</nc:PersonGivenName>
	</nc:PersonName>
</nc:Person>

As with associations, nc:RoleOfPerson can also simply contain the Person-specific information. So it can contain nc:PersonName instead of referencing an object elsewhere:

<j:CrashPerson>
	<nc:RoleOfPerson>
		<nc:PersonName>
			<nc:PersonGivenName>Peter</nc:PersonGivenName>
		</nc:PersonName>
	</nc:RoleOfPerson>
	<j:CrashPersonInjury>  
		<nc:InjuryDescriptionText>Broken Arm</nc:InjuryDescriptionText>  
	</j:CrashPersonInjury>
</j:CrashPerson>

As with associations, the choice comes down to a balance between how often the Person is used, whether you care about duplicate data in an exchange, and implementation effort.

The JSON-LD versions are, again, similar to the ones for associations. Here the nc:RoleOfPerson object has an @id that matches the one in the nc:Person object:

"j:CrashPerson": {
	"nc:RoleOfPerson": {
		"@id": "#P01"
	},
	"j:CrashPersonInjury": {
		"nc:InjuryDescriptionText" : "Broken Arm"
	}
},

"nc:Person": {
	"@id": "#P01",
	"nc:PersonName": {
		"nc:PersonGivenName": "Peter"
	}
}

And, again as with associations, you can simply include the Person-specific information inside nc:RoleOfPerson:

"j:CrashPerson": {
	"nc:RoleOfPerson": {
		"nc:PersonName": {
			"nc:PersonGivenName": "Peter"
		}
	},
	"j:CrashPersonInjury": {
		"nc:InjuryDescriptionText" : "Broken Arm"
	}
}

The choices here are also a balance, although inclusion will likely be more attractive to most JSON developers.

Artifacts

Roles
Mapping Spreadsheets

Code Tables

Codes help ensure accurate information
Codes are, essentially, strings, simple data
A few in NIEM are integers
Elements defined in the domains
Types are often defined in their own namespaces
NIEM wraps them in a complex type in order to apply some attributes needed for infrastructure
- Which we will need in the next section…
Examples:
- j:InjurySeverityCode (SSGT/Wayfarer)
  - In the SSGT, the actual codes are viewable on the page for the base simple type, e.g. aamva_d20:AccidentSeverityCodeSimpleType
  - In Wayfarer, there should be a link on the element page bringing up the codes in a separate window, e.g. aamva_d20:AccidentSeverityCodeSimpleType
- Anything ending in "Code" (SSGT/Wayfarer)
Codes nearly always have a text alternative

Schemas

j:InjurySeverityCode is a code table, with its codes defined in another namespace, the one for AAMVA. Here's the schema for the element:

<xs:element name="InjurySeverityCode" type="aamva_d20:AccidentSeverityCodeType" substitutionGroup="nc:InjurySeverityAbstract" nillable="true">
	<xs:annotation>
		<xs:documentation>A severity of an injury received by a person, such as in a traffic accident or crash.</xs:documentation>
	</xs:annotation>
</xs:element>

The actual codes are defined in a pair of types. The first, aamva_d20:AccidentSeverityCodeSimpleType, defines the actual codes and their definitions, one per enumeration below:

<xs:simpleType name="AccidentSeverityCodeSimpleType">
	<xs:annotation>
		<xs:documentation>A data type for severity levels of an accident.</xs:documentation>
	</xs:annotation>
	<xs:restriction base="xs:token">
		<xs:enumeration value="1">
			<xs:annotation>
				<xs:documentation>Fatal Accident</xs:documentation>
			</xs:annotation>
		</xs:enumeration>
		<xs:enumeration value="2">
			<xs:annotation>
				<xs:documentation>Incapacitating Injury Accident</xs:documentation>
			</xs:annotation>
		</xs:enumeration>
		<xs:enumeration value="3">
			<xs:annotation>
				<xs:documentation>Non-incapacitating Evident Injury</xs:documentation>
			</xs:annotation>
		</xs:enumeration>
		<xs:enumeration value="4">
			<xs:annotation>
				<xs:documentation>Possible Injury Accident</xs:documentation>
			</xs:annotation>
		</xs:enumeration>
		<xs:enumeration value="5">
			<xs:annotation>
				<xs:documentation>Non-injury Accident</xs:documentation>
			</xs:annotation>
		</xs:enumeration>
		<xs:enumeration value="9">
			<xs:annotation>
				<xs:documentation>Unknown</xs:documentation>
			</xs:annotation>
		</xs:enumeration>
	</xs:restriction>
</xs:simpleType>

This simple type is then wrapped in a complex type, aamva_d20:AccidentSeverityCodeType, to add infrastructure attributes that we've seen used in associations and roles. The two types are grouped together in the aamva_d20 namespace so they can be governed by AAMVA without needing to change the j domain:

<xs:complexType name="AccidentSeverityCodeType">
	<xs:annotation>
		<xs:documentation>A data type for severity levels of an accident.</xs:documentation>
	</xs:annotation>
	<xs:simpleContent>
		<xs:extension base="aamva_d20:AccidentSeverityCodeSimpleType">
			<xs:attributeGroup ref="structures:SimpleObjectAttributeGroup"/>
		</xs:extension>
	</xs:simpleContent>
</xs:complexType>

Instance Documents

The code is what shows up in the instance document. The longer definition does not. If you want to know that "3" means "Non-incapacitating Evident Injury," you need to refer to the schemas. Schema validation will verify that the value is one of the enumerations. Here's the XML:

<j:CrashPersonInjury>
	<nc:InjuryDescriptionText>Broken Arm</nc:InjuryDescriptionText>
	<j:InjurySeverityCode>3</j:InjurySeverityCode>
</j:CrashPersonInjury>

And here's the JSON. Note that NIEM does not yet support JSON Schema, so there's no means for validating the value of the code short of writing your own JSON Schema:

"j:CrashPersonInjury": {
	"nc:InjuryDescriptionText": "Broken Arm",
	"j:InjurySeverityCode": "3"
}

Artifacts

Code Tables
Mapping Spreadsheets

Metadata

Metadata is Data about Data. What does that mean? Here's an example:

Data About Jett

Name: Jett
Sex: Female
Weight: 8 pounds

Data About Data About Jett, e.g. Her Weight

Reporting Organization: Godspeed Animal Care
Reporting Person: Dr. Dudley
Reported Date: 2021-06-10
Last Verified Date: 2021-12-17

Object Reference Metadata

Objects reference the metadata objects that apply to them
An object can reference more than one metadata object
More than one object can reference the same metadata object
Example: j:Metadata (containing j:CriminalInformationIndicator) (SSGT/Wayfarer)

Connecting Metadata to Objects

Connections	Diagram
Easy to apply multiple metadata objects to one object
Easy to apply same metadata to many objects
Many-to-many relationships can get messy

Powerful, but can be difficult to implement.

Should be simpler in NIEM 6.

Schemas

The j:Metadata object is of j:MetadataType. The appinfo:appliesToTypes attribute is information for tools to potentially use.

<xs:element name="Metadata" type="j:MetadataType" nillable="true" appinfo:appliesToTypes="structures:AssociationType structures:ObjectType">
	<xs:annotation>
		<xs:documentation>Information that further qualifies the kind of data represented.</xs:documentation>
	</xs:annotation>
</xs:element>

There's nothing special about j:MetadataType. It's just an object holding some other objects, in this case a couple booleans indicators, j:CriminalInformationIndicator and j:IntelligenceInformationIndicator. It's based on structures:MetadataType, which just adds the linking infrastructure we've seen with associations and roles:

<xs:complexType name="MetadataType">
	<xs:annotation>
		<xs:documentation>A data type for information that further qualifies the kind of data represented.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:MetadataType">
			<xs:sequence>
				<xs:element ref="j:CriminalInformationIndicator" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:IntelligenceInformationIndicator" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

Instance Documents

<j:Metadata structures:id="JMD01">
	<j:CriminalInformationIndicator>true</j:CriminalInformationIndicator>
</j:Metadata>

<j:Charge structures:id="CH01" structures:metadata="JMD01">
	<j:ChargeDescriptionText>Furious Driving</j:ChargeDescriptionText>
	<j:ChargeFelonyIndicator>false</j:ChargeFelonyIndicator>
</j:Charge>

JSON-LD doesn't support a specific metadata link, so for JSON we just use @id and rely on the term "Metadata".

"j:Metadata" : {
	"@id": "#CH01",
	"j:CriminalInformationIndicator": true
},

"j:Charge": {
	"@id": "#CH01",
	"j:ChargeDescriptionText": "Furious Driving",
	"j:ChargeFelonyIndicator": false
}

Artifacts

Metadata
Mapping Spreadsheets

Combining Domains (Augmentations)

Sometimes you just want to add properties from other domains to an object without making a special kind of thing
Augmentations are bags of stuff
Augmentation Points are hooks onto which to hang the bags of stuff
Examples:
- nc:Incident (SSGT)
- j:IncidentAugmentation (SSGT)
Augmentations are an easy way to extend objects to meet your exchange needs
- We'll add an email address to a driver license using an augmentation we'll build
- j:DriverLicense (SSGT)
- j:DriverLicenseAugmentationPoint (SSGT)

Does driver's license have what we need? Does it have an email address?

Try searching for "drivers license":
- SSGT
- Wayfarer
Try searching for "driver license", singular:
- SSGT
- Wayfarer
- It's usually a good idea to drop pluralization in search terms
Check to see if it has an email address. (Spoiler, it doesn't.)
But we should learn its structure while we're here:
- Check what can contain it, specifically j:CrashDriver
- Follow up to j:CrashVehicle
- Follow up to j:Crash

Follow a similar process for nc:ContactEmailID and nc:ContactInformation. Learn about the object, not just the individual elements.

The SSGT is best for learning overall structure, so check out j:Crash and nc:ContactInformation there, too!

(Why not j:CrashDriverLicense? It doesn't go inside a j:CrashDriver. Why? The domain submitted it like this. They'll review this for the next release.)

Schemas

j:DriverLicense is defined to be of j:DriverLicenseType:

<xs:element name="DriverLicense" type="j:DriverLicenseType" nillable="true">
	<xs:annotation>
		<xs:documentation>A license issued to a person granting driving privileges.</xs:documentation>
	</xs:annotation>
</xs:element>

j:DriverLicenseType contains j:DriverLicenseAugmentationPoint as a hook for a augmentation bags:

<xs:complexType name="DriverLicenseType">
	<xs:annotation>
		<xs:documentation>A data type for a license issued to a person granting driving privileges.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="j:DriverLicenseBaseType">
			<xs:sequence>
				<xs:element ref="j:DriverLicenseEnhancedIndicator" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseCommercialClassAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseCommercialStatusAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseNonCommercialClassText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseNonCommercialStatusAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicensePermit" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicensePermitQuantity" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseWithdrawal" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseWithdrawalPendingIndicator" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseCardIdentification" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseRestriction" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseEndorsement" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:DriverLicenseAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

While NIEM does have some augmentations pre-defined, they're particularly useful for adding new objects, or just putting existing NIEM objects somewhere else. Here we do the latter, creating a bag called LicenseAugmentation with nc:ContactInformation inside.

ext: is the nickname we're going to use for our own extension namespace, where we'll create our own objects. This is a separate file we create. In it we build new objects, based on existing NIEM objects. We'll learn more about how these files fit together when we get to creating schemas.

We can now "hang" it on the j:DriverLicenseAugmentationPoint hook with nothing more than a substitutionGroup attribute:

<xs:complexType name="LicenseAugmentationType">
	<xs:annotation>
		<xs:documentation>
			A data type for additional information about a license.
		</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:AugmentationType">
			<xs:sequence>
				<xs:element ref="nc:ContactInformation"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

<xs:element name="LicenseAugmentation" type="ext:LicenseAugmentationType" substitutionGroup="j:DriverLicenseAugmentationPoint">
	<xs:annotation>
		<xs:documentation>
			Additional information about a license.
		</xs:documentation>
	</xs:annotation>
</xs:element>

Our new ext:LicenseAugmentationType is based on the built-in structures:AugmentationType. That base type merely adds in infrastructure support for linking objects together:

<xs:complexType name="AugmentationType" abstract="true">
	<xs:annotation>
		<xs:documentation>A data type for a set of properties to be applied to a base type.</xs:documentation>
	</xs:annotation>
	<xs:attribute ref="structures:id"/>
	<xs:attribute ref="structures:ref"/>
	<xs:attribute ref="structures:uri"/>
	<xs:anyAttribute namespace="urn:us:gov:ic:ism urn:us:gov:ic:ntk" processContents="lax"/>
</xs:complexType>

The resulting XML Instance Document looks like:

<j:DriverLicense>
	<j:DriverLicenseCardIdentification>
		<nc:IdentificationID>A1234567</nc:IdentificationID>
	</j:DriverLicenseCardIdentification>
	<ext:LicenseAugmentation>
		<nc:ContactInformation>
			<nc:ContactEmailID>peter@wimsey.org</nc:ContactEmailID>
		</nc:ContactInformation>
	</ext:LicenseAugmentation>
</j:DriverLicense>

The equivalent JSON-LD would be:

"j:DriverLicense": {
	"j:DriverLicenseCardIdentification": {
		"nc:IdentificationID": "A1234567"
	},
	"ext:LicenseAugmentation": {
		"nc:ContactInformation": {
			"nc:ContactEmailID": "peter@wimsey.org"
		}
	}
}

Artifacts

Augmentations
Mapping Spreadsheets

External Standards

NIEM supports external standards, if they’re XML
Wraps them in NIEM-conformant adapters
Example:
- NIEM: nc:LocationGeospatialCoordinateAbstract (SSGT/Wayfarer)
- NIEM-conformant adapter: geo:LocationGeospatialPoint (SSGT)
- External standard: gml:Point (SSGT)
You’ll probably never use this, but should know it’s there
SSGT supports them; Wayfarer does not

Schemas

LocationType contains a nc:LocationGeospatialCoordinateAbstract:

<xs:complexType name="LocationType">
	<xs:annotation>
		<xs:documentation>A data type for geospatial location.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:ObjectType">
			<xs:sequence>
				<xs:element ref="nc:LocationAddressAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationArea" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationCategoryAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationContactInformation" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationDescriptionText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationDirectionsText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationGeospatialCoordinateAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationHeightAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationIdentification" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationLandmarkAbstract" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationLocale" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationMapLocation" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationName" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationPart" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationRangeDescriptionText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationRelativeLocation" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationSurroundingAreaDescriptionText" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="nc:LocationAugmentationPoint" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

nc:LocationGeospatialCoordinateAbstract is the head of a substitution group:

<xs:element name="LocationGeospatialCoordinateAbstract" abstract="true">
	<xs:annotation>
		<xs:documentation>A data concept for a geospatial location.</xs:documentation>
	</xs:annotation>
</xs:element>

One of the members of that substitution group is geo:LocationGeospatialPoint, which is an adapter to the external standard:

<xs:element name="LocationGeospatialPoint" type="geo:PointType" substitutionGroup="nc:LocationGeospatialCoordinateAbstract" nillable="true">
	<xs:annotation>
		<xs:documentation>A 2D or 3D geometric point. A gml:Point is defined by a single coordinate tuple.
			The direct position of a point is specified by the gml:pos element which is of
			type gml:DirectPositionType.</xs:documentation>
	</xs:annotation>
</xs:element>

Its type is geo:PointType. We're still in NIEM, but gml:Point isn't:

<xs:complexType name="PointType" appinfo:externalAdapterTypeIndicator="true">
	<xs:annotation>
		<xs:documentation>A data type for a 2D or 3D geometric point. A gml:Point is defined by a single
			coordinate tuple. The direct position of a point is specified by the gml:pos element which is
			of type gml:DirectPositionType.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:ObjectType">
			<xs:sequence>
				<xs:element ref="gml:Point" minOccurs="1" maxOccurs="1">
					<xs:annotation>
						<xs:documentation>A gml:Point is defined by a single coordinate tuple.
							The direct position of a point is specified by the gml:pos element
							which is of type gml:DirectPositionType.</xs:documentation>
					</xs:annotation>
				</xs:element>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

And here is the gml:Point object from the external standard.

<element name="Point" type="gml:PointType" substitutionGroup="gml:AbstractGeometricPrimitive">
	<annotation>
		<documentation>A Point is defined by a single coordinate tuple. The direct position of a point is specified by the pos element which is of type DirectPositionType.</documentation>
	</annotation>
</element>

<complexType name="PointType">
	<complexContent>
		<extension base="gml:AbstractGeometricPrimitiveType">
			<sequence>
				<choice>
					<element ref="gml:pos"/>
					<element ref="gml:coordinates"/>
				</choice>
			</sequence>
		</extension>
	</complexContent>
</complexType>

The resulting XML instance looks like this:

<nc:Location>
	<geo:LocationGeospatialPoint>
		<gml:Point gml:id="point1">
			<gml:pos srsName="urn:ogc:def:crs:EPSG::4326" srsDimension="2">40.689167 -74.044444</gml:pos>
		</gml:Point>
	</geo:LocationGeospatialPoint>
</nc:Location>

It doesn't make much sense to try and represent this in JSON in a NIEM-conformant way. NIEM doesn't (can't) provide rules for converting non-NIEM XML to JSON. This was auto-generated with an XML editor:

"nc:Location": {
	"geo:LocationGeospatialPoint": {
		"gml:Point": {
			"gml:id": "point1",
			"gml:pos": {
				"srsName": "urn:ogc:def:crs:EPSG::4326",
				"srsDimension": 2,
				"#text": "40.689167 -74.044444"
			}
		}
	}
}

Artifacts

External Standards
Mapping Spreadsheets

Creating New Objects

Creating Simple Data Elements

Base them on whichever XML Schema data type is appropriate
- They're all in the niem-xs schema
Follow the flowchart for help
Base on the niem-xs adapters if you need referencing
- Folks generally do this anyway, for consistency
Follow the naming format of existing NIEM elements of that type, especially regarding the last term, e.g. “Text”, “Code”, etc.

Create PersonDefenestrationIndicator using the existing NIEM type niem-xs:boolean. By giving it j:CrashPersonAugmentationPoint as a substitution group head, it can go inside of the j:CrashPerson:

<xs:element name="PersonDefenestrationIndicator" type="niem-xs:boolean"
	substitutionGroup="j:CrashPersonAugmentationPoint">
	<xs:annotation>
		<xs:documentation>True if this person was thrown through a window; false otherwise.
		</xs:documentation>
	</xs:annotation>
</xs:element>

The resulting XML instance is then:

<j:CrashPerson>
	<nc:RoleOfPerson structures:ref="P01" xsi:nil="true"/>
	<j:CrashPersonInjury structures:metadata="PMD01 PMD02">
		<nc:InjuryDescriptionText>Broken Arm</nc:InjuryDescriptionText>
		<j:InjurySeverityCode>3</j:InjurySeverityCode>
	</j:CrashPersonInjury>
	<ext:PersonDefenestrationIndicator>false</ext:PersonDefenestrationIndicator>
</j:CrashPerson>

And the equivalent JSON-LD is:

"j:CrashPerson": {
	"nc:RoleOfPerson": {
	  "@id": "#P01"
	},
	"j:CrashPersonInjury": {
	  "nc:InjuryDescriptionText": "Broken Arm",
	  "j:InjurySeverityCode": "3",
	},
	"ext:PersonDefenestrationIndicator": "false"
}

Note the @id that links to an identical @id in the nc:Person object.

For a more complicated example, we can create the ext:PrivacyCode element along with a ext:PrivacyMetadata to hold it.

The actual codes are defined in the simple type, ext:PrivacyCodeSimpleType. Each enumeration defines a code. The documentation tags provide a longer text description of the code:

<xs:simpleType name="PrivacyCodeSimpleType">
	<xs:annotation>
		<xs:documentation>A data type for a code representing a kind of
			property.</xs:documentation>
	</xs:annotation>
	<xs:restriction base="xs:token">
		<xs:enumeration value="PII">
			<xs:annotation>
				<xs:documentation>Personally Identifiable Information</xs:documentation>
			</xs:annotation>
		</xs:enumeration>
		<xs:enumeration value="MEDICAL">
			<xs:annotation>
				<xs:documentation>Medical Information</xs:documentation>
			</xs:annotation>
		</xs:enumeration>
	</xs:restriction>
</xs:simpleType>

The complex type ext:PrivacyCodeType is then extended from ext:PrivacyCodeSimpleType. All this does is add infrastructure attributes to allow things of this type to refer to other elements, or be referred to in turn:

<xs:complexType name="PrivacyCodeType">
	<xs:annotation>
		<xs:documentation>A data type for a code representing a kind of
			property.</xs:documentation>
	</xs:annotation>
	<xs:simpleContent>
		<xs:extension base="ext:PrivacyCodeSimpleType">
			<xs:attributeGroup ref="structures:SimpleObjectAttributeGroup"/>
		</xs:extension>
	</xs:simpleContent>
</xs:complexType>

Then ext:PrivacyCode is created to be of ext:PrivacyCodeType:

<xs:element name="PrivacyCode" type="ext:PrivacyCodeType">
	<xs:annotation>
		<xs:documentation>A code representing a kind of property.</xs:documentation>
	</xs:annotation>
</xs:element>

In this exchange, this code is metadata to be applied to other objects, so we can create ext:PrivacyMetadata to hold it:

<xs:element name="PrivacyMetadata" type="ext:PrivacyMetadataType">
	<xs:annotation>
		<xs:documentation>Metadata about Privacy.</xs:documentation>
	</xs:annotation>
</xs:element>
<xs:complexType name="PrivacyMetadataType">
	<xs:annotation>
		<xs:documentation>A data type for metadata about Privacy.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:MetadataType">
			<xs:sequence>
				<xs:element ref="ext:PrivacyCode" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

Instance Documents

The resulting XML instance document is:

<ext:PrivacyMetadata structures:id="PMD01">  
	<ext:PrivacyCode>PII</ext:PrivacyCode>  
</ext:PrivacyMetadata>  

<ext:PrivacyMetadata structures:id="PMD02">  
	<ext:PrivacyCode>MEDICAL</ext:PrivacyCode>  
</ext:PrivacyMetadata>

<j:CrashPerson>
	<nc:RoleOfPerson structures:ref="P01" xsi:nil="true"/>
	<j:CrashPersonInjury structures:metadata="PMD01 PMD02">
		<nc:InjuryDescriptionText>Broken Arm</nc:InjuryDescriptionText>
		<j:InjurySeverityCode>3</j:InjurySeverityCode>
	</j:CrashPersonInjury>
	<ext:PersonDefenestrationIndicator>false</ext:PersonDefenestrationIndicator>
</j:CrashPerson>

An equivalent JSON document would be:

"j:CrashPerson": {
	"nc:RoleOfPerson": {
	  "@id": "#P01"
	},
	"j:CrashPersonInjury": {
	  "nc:InjuryDescriptionText": "Broken Arm",
	  "j:InjurySeverityCode": "3",
	  "ext:PrivacyCode": [
		"PII", "MEDICAL"
	  ]
	},
	"ext:PersonDefenestrationIndicator": "false"
}

Instead of linking to two separate metadata objects, we've embedded those into the j:CrashPeson.

Artifacts

Creating New Objects - Simple Data
Mapping Spreadsheets

Creating Complex Objects

If you’re making a special kind of existing NIEM type:
- Use that type as a base to extend from and add things to it, or
- Create an Augmentation to hold your new things
If you’re starting from scratch:
- Use structures:ObjectType as a base to extend from and add things to it

Here we're making the root element that will hold everything else. We create CrashDriverInfoType, basing it on structures:ObjectType so that it's just an empty object.

To that empty object, we add all the major objects in our exchange, nc:Person, j:Crash, and j:Charge. We also add a j:PersonChargeAssociation which lets us link together people and charges. Finally, we add a couple metadata object, the built-in j:Metadata, and ext:PrivacyMetadata, which we create in the extension schema and is in the prior example.

<xs:element name="CrashDriverInfo" type="exch:CrashDriverInfoType">
	<xs:annotation>
		<xs:documentation>A collection of information about the driver of a vehicle in a crash.</xs:documentation>
	</xs:annotation>
</xs:element>

<xs:complexType name="CrashDriverInfoType">
	<xs:annotation>
		<xs:documentation>A data type for a collection of information about the driver of a vehicle in a crash.</xs:documentation>
	</xs:annotation>
	<xs:complexContent>
		<xs:extension base="structures:ObjectType">
			<xs:sequence>
				<xs:element ref="nc:Person" maxOccurs="unbounded"/>
				<xs:element ref="j:Crash"/>
				<xs:element ref="j:PersonChargeAssociation" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:Charge" minOccurs="0" maxOccurs="unbounded"/>
				<xs:element ref="j:Metadata" minOccurs="0"/>
				<xs:element ref="ext:PrivacyMetadata" minOccurs="0" maxOccurs="unbounded"/>
			</xs:sequence>
		</xs:extension>
	</xs:complexContent>
</xs:complexType>

Another example is when we created ext:LicenseAugmentation and ext:LicenseAugmentationType when talking about augmentations.

Adding New Content to the Exchange

To summarize, there are two major ways to add new content to an exchange. We've seen them both above.

If you're already creating a new complex object and type, like CrashDriverInfo and CrashDriverInfoType above, you can simply add new elements to the new type, as we did with ext:PrivacyMetadata. This is called "concrete extension."

But if you're not already creating the new type for other reasons, you should use augmentations. We used added j:CrashPersonAugmentationPoint to our exchange and used it as a hook on which we hung ext:PersonDefenestrationIndicator.

Problems with Concrete Extensions

The issue with concrete extensions is that if the extension is happening deep inside an object, all the surrounding objects will also need to be extended. For example, if instead of using j:DriverLicenseAugmentationPoint, suppose we extended j:DriverLicenseType, making a new ext:DriverLicenseType to hold our new object. Now we need a new element for it, ext:DriverLicense.

But j:CrashDriver doesn't hold an ext:DriverLicense, so we need to make a new ext:CrashDriverType and ext:CrashDriver that can hold an ext:DriverLicense.

But j:CrashVehicle doesn't hold a ext:CrashDriver, so we need to make a new ext:CrashVehicleType and ext:CrashVehicle that can hold an ext:CrashDriver.

But j:Crash doesn't hold an ext:CrashVehicle, so we need to make a new ext:CrashType and ext:Crash that can hold ext:CrashVehicle.

That's a lot of extra work and muddies the semantics of the elements.

Artifacts

Creating New Objects - Complex Objects
Mapping Spreadsheets

Mapping Completed!

Crash Driver Report Complete
Mapping Spreadsheets

Started With	Ended With

Build and Validate

Conformance
Subsets
Extension and Exchange Schemas
How things fit together

NIEM Conformance

Follow the rules in the Naming and Design Rules (NDR)
Follow the rules in the IEPD Spec
Many NDR rules exist as Schematron
- Can check these via the Conformance Testing Assistant (ConTesA)
- Can run the Schematron directly oXygen, with a plug-in in XMLSpy

Schema Subsets

NIEM has ~13,000 elements
You don’t want them all
NIEM supports mini versions of NIEM
- With the elements / types you want
- Plus things needed to make the elements / types you want work
Use the SSGT for this! Demo Time!
MEP Builder can also help! More Demo Time!

Extension Schema(s)

Create new elements for your exchange
Emulate how NIEM does it
Utilize Augmentations to add your new objects to existing NIEM objects
- Concrete extension is an alternative
Can have multiple extension schemas
Some folks put code tables into a separate extension schema

Exchange Schema

Entry point to the exchange
Defines the root element of the exchange
Some put the definition for the root element here
Some put that in the extension schema
Some lump exchange and extension together
An exchange with multiple messages can have multiple exchange schemas, one per
- Can share a common extension
- Or not

How Schemas Fit Together

Help with Schemas

Use a good validating XML editor
- XMLSpy
  - Windows
- oXygen
  - Windows, macOS, Linux
Declare a conformance target
- Just use the same one I am
Declare and import all schemas used
- Watch out for schemas incorporated only via substitution groups
- Group heads don’t know who the members are
- Still need to declare and import those

Example Instances

Required
Validating against your schemas ensures your intent
Some XML editors can create them from schemas
- But you’ll always need to tweak those
JSON instances are more of a manual process because NIEM doesn't support JSON Schema
- It's in the works
Other tools are out there…

Tips and Tricks

Christina’s oXygen Snippets
- oXygen only
Use Schematron to check your work as you go
- oXygen
- XMLSpy with plug-in
MEP Builder
- Work on mapping spreadsheets
- Generate subsets
Tom’s Mapping Spreadsheet Linter

Assemble and Document

IEPD Catalog
- Metadata about the Message Spec / IEPD
- What file is what
Readme
Changelog
Conformance Assertion
Mapping Spreadsheet
Master Documentation

Master Documentation

Just a Word document
- Or another document format, e.g. Markdown, PDF, etc.
A NIEM Message Spec / IEPD master document should (at a minimum) describe:
- Message Spec / IEPD purpose
- Scope
- Business value
- Exchange information
- Senders/receivers
- Interactions
- References to other documentation, if applicable
Don’t paste schemas into them!

Message Spec / IEPD Assembly

Message Specs / IEPDs are just ZIP files full of artifacts
Can organize how you like
- The IEPD Catalog is the guide to your organization
- Generally, keep it simple
MEP Builder will help with this (phase one available now)

Publishing

Existing repositories are out of date
- IEPD Clearinghouse
- Work With IEPDs (currently not working)
New repositories are coming
- Restricted (brand new)
  - Warfighting Mission Area-Architecture Federation and Integration Portal (WMA-AFIP)
  - If you don't know what this is, you probably don't have access
- Unrestricted (coming later)

Implement

Remember that NIEM is just the payload
- It’s just plain old XML
  - Well, okay, with referencing
- Or it’s just plain JSON
  - Well, okay, with JSON-LD
Remember the scope of NIEM
- Things outside the payload aren't NIEM
- You don’t have to do anything special for NIEM outside NIEM’s scope

Skill level to Implement NIEM

Developers with the skills to design and implement a data exchange can learn the NIEM approach in a matter of hours
Developer training is available
Plenty of example IEPDs to follow
- Hard to find right now
- Will get better
Don’t start from scratch, leverage shared IEPDs
The NIEM technical specifications are complex, however
- Most developers do not need to read them
- Free tools can perform most of the conformance checking
  - Schematron directly
  - Conformance Testing Assistant (ConTesA)

Resources

Documentation

NIEM Releases
- http://niem.github.io/niem-releases/
NIEM Specifications
- http://niem.github.io/reference/specifications/
NIEM JSON Spec
- https://niem.github.io/NIEM-JSON-Spec/v5.0/
Materials from this course:
- https://github.com/NIEM/NIEM-Training
NIEM.gov Contact Page
- https://www.niem.gov/contact

Tools

SSGT:
- https://tools.niem.gov/niemtools/ssgt/index.iepd
Conformance Testing Assistant (ConTesA)
- https://tools.niem.gov/contesa/
NDR Schematron
- https://niem.github.io/reference/tools/oxygen/ndr/
MEP Builder
- https://sourceforge.net/projects/niem-mep-builder/
Wayfarer:
- http://niem5.org/wayfarer/
oXygen Snippets
- https://niem.github.io/reference/tools/oxygen/snippets/
Mapping Spreadsheet Linter / Schema Generator
- http://niem5.org/linter/

Repositories

IEPD Clearinghouse
- https://bja.ojp.gov/program/it/policy-implementation/clearinghouse
IEPD Repository (Work with IEPDs) - currently inoperative
- https://tools.niem.gov/niemtools/iepdt/index.iepd

Files

Master NIEM Training Document.md

Latest commit

History

Master NIEM Training Document.md

File metadata and controls

Master NIEM Training Document

Introduction

Purpose

Supporting Documents

Sample Message Specification

Agenda

Logistics and Background

Introduction to NIEM

What is NIEM? - Framework

What is NIEM? – Interop Problem

What is NIEM? And what not?

What is NIEM? – Exchanging Data and Components

What is NIEM? – Simplified Exchanges

Scope of NIEM

Scope of NIEM - Open Source Interconnection (OSI) model

Bottle of Liquid

NIEM Harmonization and Organization

Message Spec / IEPD Overview

Message Spec Process

Message Spec / IEPD Artifacts - Documentation

Message Spec / IEPD Artifacts - Definitional

Scenario Planning

Existing Documentation

Analyze Requirements

UML Class Diagrams

Business Oriented Class Diagram

NIEM Oriented Class Diagram

UML Tools

Business Rules

The Process from Here On…

XML Schema in a Nutshell

Map and Model

Introduction to Mapping

NIEM Mapping Template

Simple Training Spreadsheet

Tom’s Custom Mapping Spreadsheet

Basics of Searching NIEM

Mapping and NIEM Technical Details

Understanding NIEM Objects

XML Schema

What About JSON?

Native Properties

Artifacts

Substitution Groups

Schemas

Instance Documents

Artifacts

An aside about Namespaces

Inherited Properties

Schemas

Instance Documents

Artifacts

Linking Things Together

NIEM Structural "Layers"

Referencing - XML Schema

Referencing - JSON-LD

Associations

Schemas

Instance Documents

Roles

Searching for Injured Person

Roles - Not Special Kinds of People

Roles – Allows an Object to Play Multiple Roles

Roles - How They Work

Schemas

Instance Documents

Artifacts

Code Tables

Schemas

Instance Documents

Artifacts

Metadata

Data About Jett

Data About Data About Jett, e.g. Her Weight