AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR - PowerPoint PPT Presentation

Loading...

PPT – AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR PowerPoint presentation | free to download - id: 670926-ZWQ0M



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR

Description:

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR ADE System Architecture September 4, 2008 – PowerPoint PPT presentation

Number of Views:36
Avg rating:3.0/5.0
Slides: 109
Provided by: nwsNoaaG95
Learn more at: http://www.nws.noaa.gov
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR


1
AWIPS Continuous Technology Refresh (CTR)AWIPS
Software CTR
  • ADE System Architecture
  • September 4, 2008

2
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

3
System Concept AWIPS-II Architecture Data
Fusion and Visualization SOA Framework
Architecture Introduction
  • Requirements Vision Drives Architecture
  • Focus on ilities drives new architecture
  • Features and capabilities get generalized into
    reusable patterns
  • Customer TIMs give priority to capabilities
  • Architecture Framework Vision
  • Create a new low cost framework for hosting a
    full range of environmental services including
    thick client visualization
  • The framework will scale down to a small laptop
    and up to clusters of enterprise servers without
    software change
  • The framework will be based on highly reusable
    design patterns that maximize reuse, has datatype
    independence, and fast adaptability
  • Open source is leveraged to maximize reuse

4
AWIPS-II Architecture Definitions
Architecture Introduction
  • ADE AWIPS Development Environment, source code
    to execution framework enterprise development kit
    including tools
  • SOA, End points, I/O Routing, Transforms Service
    Orientated Architecture where system capability
    is available at stateless endpoints
  • Canonical XML Well formed XML that follows high
    level rules
  • Patterns Implements a design solution that
    solves a problem that occurs many times
  • Technical Reference Architecture A physical
    software execution framework
  • JMS, JMX Java Messaging System (API), Java
    Management Extensions
  • CAVE Common AWIPS Visualization Environment
    using Eclipse RCP
  • SEDA Serial Event Driven Architecture

5
AWIPS-II Architecture Concept Architecture
Framework Implementation
Architecture Introduction
  • Framework Implementation Integrated several
    best of breed open source projects with a set of
    advanced enterprise patterns to create a highly
    extendable framework.
  • Patterns Implemented in Pure Java code (Reuse
    Example ProductSrv and AutoBldSrv use uEngine)
  • Open Source is primary for re-use
  • 15 major open source projects integrated
  • Version controlled with CM baseline, libraries
    part of run env.
  • Leverage Internet community for core
    infrastructure
  • Standards compliant, rapid evolution
  • Free, large body of public expertise
  • Open source libraries controlled by putting them
    in the CM compile library and deploying them to
    the runtime environment
  • Packaged together into the ADE which contains
    everything from the Source Code repository to the
    execution environment including operator Clients

6
AWIPS-II Problem Definition User NeedsDrives
Architecture Towards Advanced Solutions
Architecture Introduction
Totally Non-Propriety Solution (Open Source)
First Person Shooter Video Game
Weather Edge User Visualization
Quickly Adaptable New Science Data Types
Accelerate Innovation Speed of Deployment
Google Earth
Sat Com
Increasing Data Interactions
CAD Systems
Situational Awareness Visualization
Con-Ops Flexibility 164 Unique Op Centers
Increasing Data Rates
High Data Rates With Gaming Style Interaction
7
AWIPS-II Development Sequence, T09 is GFE 3
to 6 month fixed price task orders
Architecture Introduction
2006
2007
2008
2009
2010
Today
AWIPS II Development
T06
T04
T05
T03
R D
2 Week Cycles Stability Testing Full Data Load
Architecture Development
T08
Clean Sheet Start
Wx Workstation
T09
AWIPS-II Release 1
Forecast Editor
T010
Hydrology
T011
Comm
Deployment
Spiral Development with Interim Deliveries
8
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

9
SOA Architecture Logical Layered ViewLayers
Separated By Simple APIs
Architecture Overview
Client/Presentation Services
JMX
CAVE
External Programs
Mission Services Layer
Mbean
ProductSrv
NotifySrv
IngestSrv
VtecSrv
Hydro Models
AutoBldSrv
AdapterSrv
ArchiveSrv
SubscribeSrv
UtilitySrv
LAPS
IndexSrv
Security Services /Demilitarized Zone (DMZ)
Enterprise Service Bus - Communication
PurgeSrv
StagingSrv
FORTRAN/C/C Command Line Programs
Data Access Layer
Hibernate
HDF5 API
Synchronous Interface
Localization Store
Platform Layer
PostgreSQL
HDF5
Metadata Index
Data Persistence Store
Spatial Index
10
AWIPS-II ESB SOA FrameworkGeneric Core Services
Adapt Through Plug Ins
Architecture Overview
ltltEDEX Framework Architecturegtgt Services Structure
  • Core Base of Services
  • Micro Engine
  • Plug-in Framework
  • Core libraries

ltltCAVE Visualization Framework gtgt
core plug-ins
libraries
libraries
core plug-ins
Ingest Data
Disseminate
Index Data
Store Data
  • Extend to a specific set of services
  • Plug in specific libraries
  • Plug in data types, transforms scripts

Transform Data
Build Products
Collaboration
Fusion Engine
ltltTechnical Implementationgtgt Concrete Services
based on Data Types
ltltVisualization Extension gtgt
Data Types
Meta Data
Transforms
weather libraries
Vis Plugins
Visualize Data
Data
Scripts
Vis Scripts
mapping libraries
Mapping
Vis Localize
  • Service Interface To Data
  • Clean separation between data and visualization

AWIPS-II Designed for Extension to New Domains
11
AWIPS-II Plug In adaptable data fusion and data
transformation framework with Visualization
Architecture Overview
Auto Metadata Data Persistence
Plug In Data Adaptable
Configuration via XML
Gaming Visualization
CAVE
uEngine Data Transformer
Plug In Extendible
Machine to Machine
Enterprise Service Bus
Based on Advanced Design Patterns
12
AWIPS-II Designed for reuse across multiple
domains
Architecture Overview
  • Extending to new domains
  • Add Data Type Plug Ins
  • Add Libraries
  • Add Scripts
  • Add CAVE Plug Ins

AWIPS-II Architecture
Domain3
Domain 1
Domain 2

Low Cost Extension to New Domains
13
AWIPS-II Dependencies Open Source BasedFree of
all fee based COTS dependencies
Architecture Overview
Mule ESB Spring Enterprise service bus and dependency injection container for SOA services
Eclipse RCP Plug In driven visualization framework
GeoTools Enables GIS capabilities and map projection framework
JOGL Java API to OpenGL enables Gaming level visualization performance
HDF5 High performance file persistence of large data sets such as satellite and radar
JavaScript Python numPY Data transform scripting languages with high performance math library
JAVA ANT Primary programming language and software build framework
activeMQ Java messaging provider with clustering
PostgreSQL Relational database for storing Metadata from Data plug ins and spatially enables ingested data
COTS Flexibility
14
AWIPS-II High Level System Services SOA Services
Running in an ESB Container
Architecture Overview
CORE SOA Services
WAN Visible
JMX Remote Service Management Client
ADE 1.0
Mbean
Mbean
Mbean
Mbean
Mbean
IngestSrv StagingSrv
PurgeSrv
ArchiveSrv IndexSrv
ProductSrv
CAVE Visualization Client
AdapterSrv
Data Rendering
TAF Plug In
FileSystem
Meta Data Index
uEngine
Exec Adapter
CAVE Bundles
METAR Plug In
RDBMS via JDBC
Spatial Data Base
Manage Subscription
JNI Adapter
Radar Plug In
Data Interrogation
HDF5 persistence
GFE IFPServer
Satellite Plug In
CAVE Procedures
Wx Drawing
GRIB Plug In
Wx Warning
Radar All Tilts
Mbean
Hydro Visualization
UtilitySrv
Shapefile Visualization
Mbean
Collaboration
Mbean
Mbean
AutoBldSrv
NotifySrv
VtecSrv
Eclipse config XML
Localization Data
XMPP
Color Tables
uEngine
Subscription Notify
Maps and Topo
Enterprise Service Bus HTTP, JMS, Virtual
Memory, File Endpoints
Enterprise Service Bus HTTP, JMS, Virtual
Memory, File Endpoints
Services Independent of End Points
15
Requests Data for Display as a GIS Layer
Layered View of SOA Service with Data Flow
Architecture Overview
Client/Presentation Layer
Native Services Layer
ProductSrv
CAVE
SEDA Service Scaling Within Container
Platform/Infrastructure Services Layer
Input Message Canonical XML With body containing
action commands
uEngine Pattern
Enterprise Service Bus HTTP,JMS, VM, Endpoints
termQuery
fileIn
makeResponse
Output Message Canonical XML Response to Client
Containing URI references to data generated
Data Access Objects
Platform/Infrastructure Resources Layer
Persistence Repository HDF5
MetaData Index
16
Ingest Data Flow Layered View of SOA Service
Ingest at a Clustered End Point
Architecture Overview
Native Services Layer SOA ESB Services
Platform/Infrastructure Resources Layer
Queue Clustering
File Name
/awips/opt/data/sbn/ /awips/opt/data/processing
vm//IndexVMQueue
IndexSrv
StagingSrv
IngestSrv
Platform/Infrastructure Services Layer
Enterprise Service Bus HTTP,JMS, VM, Endpoints
Plugin(pluginType)
File End Point
DataLayer saveData(record) saveMetadata(record)
Data Access Objects
HDF5Dao
Platform/Infrastructure Resources Layer
Persistence Repository HDF5
MetaData Index
17
AWIPS-II Alerting Pattern UpdateAdded Observer
Interface to enable multiple users
Architecture Overview
Data Item
EDEX
DecisionTree
CAVE
CAVE Startup
Quinlan ID3 Algorithm
AlertJobStarter
Build Data URI
Menu dataURIs
URICatalog
Message Received
MenuUpdater
Data URI List
ingested dataURIs
quartz Timer
Data URI Aggregator
AvnFPS
IAlertObserver
topicedex.alerts
18
AWIPS-II Data Cube EnhancementsRedesigned data
cube to enhance performance flexibility
Architecture Overview
HDF5 Grid Repository
Volume Browser
RequestJob
PlanViewDisplay
Loader
TimeMatcher
BrowserDialog
DataCubeContainer
BrowserDataCatalog
purgeDataCube refreshDerivedParameters examineAttr
ibutes getDataRecord performTimeQuery List setget
GridTree
ltJobgt DataCubeManager
GridTree
Dynamically loaded
Python Derived Parameter
Python Derived Parameter
DerivedParameterGenerator
DerivParamScriptsLoader
19
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

20
SOA Service PatternDesign Pattern
SOA Service
  • SOA Service Vision All server side capabilities
    are available as a standard service which
    communicate using standard network protocols and
    formats.
  • Implementation All services inherit from a
    single base class that encapsulates life cycle
    and communicate details. Services are Dependency
    Injection components that are configured to
    communicate with any supported endpoint.

Hides Interface Details to Container Extracts
message Calls process() on return Reports
exceptions
AbstractSrvMBean
Registers with JMX for Remote monitoring
AsciiIngestSrvMBean
Does the work of the Service Return puts message
on outbound endpoint
21
SOA Service PatternDependency Injection -
Minimizes Coupling
SOA Service
  • Dependency Injection All AWIPS SOA services get
    there interfaces defined by XML configuration
    which maximizes service reuse and minimizes
    hardcoded coupling.

Service interface and configuration
Files Generally One file for each service
Service output endpoint Message sent when
process() executes return
Service class file reference
File endpoint wakes up Services
process() method when data is picked up
22
SOA Service PatternService Container and ESB
startup
SOA Service
  • Startup ADE 1.0 has all SOA services running in
    one container. The start script automatically
    builds the classpath from jars in lib directory.

SOA Service Container/ESB start scripts start
linux start.bat -- Windows
Sets clustering configuration
Service Configuration Files, Adding a
service Includes adding a new service config file
to the startup.
Builds list of ESB End-point and Wiring Files
Starts Mule ESB
Dependency Injection Container
23
Canonical XML Interface PatternSOA service
interfaces
SOA Service
  • Vision A simplified XML based message format
    that is independent of the endpoints. The
    message schema is flexible and is orientated
    towards the client.

Example Response Message From ProductSrv
Top Message Tag
Message Header Tag With properties
Message Body Tag
Response by URI Reference
Returned product in png image format
24
Canonical XML Interface PatternSOA service
interfaces
SOA Service
  • Reponse The body of the response is a island of
    XML, a base64 encoded binary image, or a URI
    reference for large data sets.

Example XML Response
binding.xml
AbstractResponseMessage
-fileType -dataURI -validTime
ResponseMessageError
-errorSource -errorMsg -errorCause -errorChain
ResponseMessageASCII
ResponseMessageURI
ResponseMessageInline
-filename -contents -type -station -time
-productURI
-fileName -zippedData
25
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

26
AWIPS-II SOA Data Type Plug In PatternEnables
New Data Types and Transforms
Plug In Pattern
EDEX Services
SOA Data Type Plug In
Index
Ingest
MetaData Interface
XML MetaData Def Data URI Def
Decoder Interface
MetaData Decoder
HDF5
MetaData
Storage Interface
Message Separator
Schema Creator
Alert Topic
Notify
Data Decoder
URI Generator
uEngine Tasks
uEngine Interface
Product
Extreme Adaptability Through Plug In Pattern
27
SOA Data Plug In Enables Extensibility Meta
Data, Decode, Storage, Transforms
Plug In Pattern
Enterprise Service Bus Dependency Injection
Container
SOA Data Type Plug In
Class Loader
ltltSOA Servicegtgt ProductSrv
XML Msg
Jibx
uEngine
Enables Extending the uEngine Language
Enterprise Service Bus
JEBB Python Interface
XML Msg Response
Hibernate DAL
Dynamic Meta Data Creation
HDF5 Persistence
Dynamically Extends System to New Problems
28
Plug Ins Enable Adapting to New
DataAutomatically Creates Meta Data Persistance
Plug In Pattern
  • Reference Architecture
  • IProductExtract Metadata
  • IRecord
  • IDecode
  • IStore
  • Task
  • Discovery

Plug In Package (JAR) plugin.xml metadata.xml attr
ibutes.xml Implementation Classes Binding
Classes uEngine Task Implementations
Enables new science through new Tasks
Enables new data to Be queried and visualized
Dynamically Extendible to New Data Transforms
29
Plug Ins Enable Adapting to New DataDynamically
Extends to New Data Transforms
Plug In Pattern
ESB Endpoint Config
Plug In (i.e)plugin-airep
Auto Generated HDF5 schema for Blob data
Metadata DB
HDF5 Lighting Radar Satellite Grid
Auto Generates Schema
Creates Data URI
Specific Data Type Decoding
Custom Data Transforms if necessary
Custom Data Access
Plug In Registry Table
30
SOA Plug In CM File structure
Plug In Pattern
SOA Data type Plug Ins
Plug In XML and Configuration
Meta data definition files
Plug In Implementation Classes
uEngine Task Scripts Extends ScriptTask
31
AWIPS-II T09 Services and Plug InsESB SOA Plug
In Adaptable
Plug In Pattern
Future
T09
T010 Improve
Airep (22 total) Binlightening Bufrmos Bufrua ccf
p 268 Gfe 10808 Goessounding 547 Grib modelsoundin
g 846 Obs Pirep poessounding 598 profiler
560 Radar Recco redbook 730 Satellite Sfcobs shef
3973 Taf Text warning 1115
Eclipse RCP Visualization Plug Ins
GfePurgeSrv
Bufrmos 801
SmartInit
Ingest
Staging
14 Core 8 User Interface 22 Library 14 Application
HdfHttp
GFE 26944 Redbook 619 Python Editor 122 Hydro
10039 aviation(avnFPS) 11539
Purging
VTEC
Index
Archive
Notify
Adapter
Utility
AutoBld
Product
py
GFE
CAVE
Proxy
AWIPS-II Services
Thick /Light Client Mode
/Light
Text 678
Enterprise Service Bus
SOA Data Type Plug Ins
32
SOA Plug In Tool Added to ADEAdding using the
tool to a training module
Plug In Pattern
  • New with ADE 1.0 a Plug-in Creation tool.
  • The Plug-in Creation tool is implemented as an
    Eclipse plug-in.
  • Once installed, the Plug-in Creation tool may be
    used like any other element of the Eclipse IDE.
  • The Plug-in Creation tool automates most of the
    work involved in generating the initial files for
    a data-type plug-in.
  • Automatically creates the required directory
    structure for the plug-in.
  • Generates class stubs for the required Java
    classes.
  • Generates initial configuration and build files.

33
Software CM/Build/Deploy PatternDesign pattern
Build Patterns
  • Build Vision Create a simple layered build
    system that manages component coupling and
    supports partial deployment.
  • Build Implementation Implemented in ANT as a
    series of macros and ANT extensions.
  • Jar Files for Compiling
  • Open Source Jars

deploy.xml
test.xml
build-global.properties
config.xml
compile.xml
build-local.properties
build.xml
jibx. xml
mortar.xml
javadoc.xml
34
Software CM/Build/Deploy PatternDesign pattern
(Server Side)
Build Patterns
CM Controlled Baseline (Subversion)
1) Build services components
ant build -Dcompfiledeployments/deployment.proper
ties
2) Deploy to run time environment
ant deploy
Services and Components Open Source Dependencies
Special Jars
Plugin Jar Files
35
Deployment Simplification
Build Patterns
  • Created a new GUI based installer
  • Standard Linux Server Install for Server Side
    pieces
  • Standard Linux Workstation Install for CAVE
  • Windows Workstation for CAVE
  • Procedure for clustering (Clustering Con-ops?)
  • Linux Server 1
  • Edex server
  • Jdk
  • PostgreSQL
  • Eclipse 3.3
  • ANT 1.7
  • Edex source code
  • Edex Javadoc
  • Linux Workstation
  • CAVE
  • Basemaps
  • Jdk
  • Topo
  • Win Workstation
  • CAVE
  • Basemaps
  • Jdk
  • Topo

Remote Mount
Data Partition
36
AWIPS-II Deployment Simplification
Build Patterns
  • CAVE application and EDEX services installed
    using two separate installers
  • Each installer can install all files to the
    users home directory.

37
AWIPS-II Localization PatternEnables SOA
customization of configuration
Localization Pattern
  • Localization procedure occurs at startup
  • CAVE localization is controlled through Eclipse
    Preferences
  • Extended Eclipse Preferences to use a XML data
    store
  • Different approach to data since CAVE can work
    with large data sets less subsetting is required
  • Localization is provided through two simple,
    unified interfaces
  • Configuration for the server
  • Localization for the client, with server
    synchronization capability
  • Localization provides a multi-tiered
    configuration
  • Base (standard national configuration values),
    Site for Server (any values the site chooses to
    override)
  • Base, Site and User for Client (values that the
    user overrides)

38
Workstation (CAVE) Localization PatternUses
UtilitySrv with a HTTP interface
Localization Pattern
ltltSOA Servicegtgt UtilitySrv
ESB Sync Protocal
CAVE Config
At CAVE Start Up
Log In User Name Determines User Context Site
Through Perferences
CAVE Menus
Localization Hierarchy
Color Maps
Base
Local Persistence Of Current Active Localization
Site
User
39
Localization Data Hierarchical Preferences,
Menus, Data
Localization Pattern
Base
ltconfigurationgt lttextureCardPreferencegt128lt/tex
tureCardPreferencegt lttextureMemoryPreferencegt38
4lt/textureMemoryPreferencegt ltframesPerSecondPre
ferencegt25lt/framesPerSecondPreferencegt
lttileBoundariesgtfalselt/tileBoundariesgt
ltconnectionMethodgtjmslt/connectionMethodgt
ltjmsServerAddressgttcp//localhost61616lt/jmsServer
Addressgt ltdataDirectorygt/awips/opt/data/hdf5lt/d
ataDirectorygt ltfontMagnificationgt1.0lt/fontMagni
ficationgt lt/configurationgt
Site
ltconfigurationgt ltsiteNamegtKOAXlt/siteNamegt
ltsiteFullNamegtOmahalt/siteFullNamegt
ltsiteTypegtWFOlt/siteTypegt ltdataDirectorygt/oax-aw
ips/opt/data/hdf5lt/dataDirectorygt lt/configurationgt

User
ltconfigurationgt ltfontMagnificationgt1.25lt/fontMa
gnificationgt lt/configurationgt
40
uEngine Task Execution Pattern Breaks up
execution into small reusable tasks
uEngine Pattern
  • uEngine Vision Create an execution framework
    for generating custom SOA requests on-demand.
    Customer systems can request products by script
    requests over a network. The script performs
    small general units of work that get chained
    together to produce a customer product.
  • uEngine Implementation Runs out of a SOA
    service ProductSrv that is attached for I/O to
    ESB endpoints. The uEngine executes scripts in a
    standard scripting language that has been
    extended. Current scripting language is
    JavaScript and transitioning to Python with NumPy
    for improved algorithm performance.

41
AWIPS-II uEngine Design PatternExecutes
Commands to Transform/Fuse Data
uEngine Pattern
Service Container
ltSOA Servicegt ProductSvc
Inbound Endpoint
Meta Data
XML Msg
uEngine
HDF5 Persistence
F01DE29344 D222234343
queryField queryText
Enterprise Service Bus
Outbound Endpoint
XML Msg Response
SOA PlugIns Contains Extensions
IR Color
GeoTiff
imageOut
uEngine Enables Flexible User Requests/Response
42
uEngine Scripting Changed to PythonEnables
common scripting language, and numpy benefits
uEngine Pattern
(i.e)
uEngine Automatic Script Generation Users
IFPClient
Volume Browser
PlotModelGenerator
Text Workstation
WarnGen
CatalogQuery
uEngine Script
PyProductSrv
ScriptCreator
JepPool
ScriptFactory
Jep executeScript
Velocity Template Engine
Velocity Template
Build Response
43
SOA Plug In uEngine Tasks ExtensionsEclipse IDE
Views of Plug In Implementations
uEngine Pattern
Satellite Plug In
Plug Ins Each gets built Packaged in a
separately Deployable Jar
Note Each Plug In Has a uEngine Task Package
Grib Plug In
Radar Plug In
44
uEngine Task Execution PatternEnables highly
flexible SOA request/response
uEngine Pattern
Metadata Fields Defined by SOA Plug In
ltltJava Classgtgt UEngineScript
run() setupSubscription(script)
ltltJava Classgtgt UEngineUtil
convertFromJsToJava()
Provides Interface To Scripting Language
Output Product in Response by Reference
45
uEngine Scripting OverviewEnables SOA scripting
of scientific algorithms
uEngine Pattern
  • Goal Provide uEngine scripting in a language
    suited for scientific algorithms while enabling
    high performance
  • Uses standard Python with JEBB for a bridge to
    Java
  • Uses numPy with its numerical data types and
    functions
  • Includes interface to Fortran library Linpack
    for linear algebra
  • Includes interface to BLAS for taking advantage
    of multiple processing chains in modern CPUs for
    improved performance
  • UengineScript class, simply call
    setScriptText(String javaScriptCode) and run()
  • Extend ScriptTask, Basic POJOs with an execute()
    method
  • Tasks only operate on one object (loops at the
    script level)

46
ProductSrv ChangesJavaScript uEngine Language
DB for Subscribe
uEngine Pattern
ltltSOA Servicegtgt ProductSrv
uEngineScript(script)
.run()
Y
Subscribe
subscribe(subScript,dataURI,scriptID)
N
SubscribeDAO
postgreSQL MetaData
script
subscription
End
47
AWIPS-II GIS Capabilities GeoToolsGIS Data
Types, Transforms, and Display
GIS Capability
  • DataStore and Data Formats
  • ESRI Shapefile (Read/ Write)
  • WFS Web Feature Server (OGC) (Read / Write)
  • PostGIS geometric objects for PostgreSQL
  • GML Geography Markup Language
  • GeoTIFF geo-referenced TIFF image
  • Coordinate Transformation (OpenGIS CTS)
  • Map Projections (Mercator, Transverse Mercator,
    Lambert Conformal Conic, Albers Equal Area Conic,
    Stereographic, Orthographic, )
  • Math Transform

48
Map Library Capability Map reprojectionTransfo
rm Ingested Data to Display Projection
GIS Capability
  • GeoTools builds a transformation Matrix that
    gets reused in transform operations.
  • The Transformation matrix can be build up
    through a series of transforms.
  • Used In Barnes Analysis Pattern to calculate
    distances.
  • Degrib Task uses GeoTools to dynamically define
    projection data (I.e. NCEP hur.)

CoordinateReferenceSystem GridGeometry2D BufferedI
mage
GeoTools
ReprojectedImage
49
Map Libraries Through GeoTools JTSJTS Topology
Suite Spatial Operations Java API
GIS Capability
  • Computing Spatial Relationships
  • Shape File Intersect Demonstration
  • Distance Tool Demonstration

Overlay Operations
Buffer Operations
Polygonization Quantization
50
Spatial Table Refactor Geo SpatialPostgis
extension to PostgreSQL
GIS Capability
  • Geo Spatial Enabling Data
  • Chosen Approach Create Static Spatial Tables in
    PostgreSQL
  • PostGIS extension Free, simple, high performance
  • Input / Output
  • All ingested data is spatially indexed and can be
    spatially queried
  • Can create Shape and GeoTiff output

PostgreSQL Data Base
radar_spatial
spatial_grids
spatial_obs_stations
spatial_satellite
51
Legacy Adapter Pattern SOA enable legacy
application functions
Adapter Pattern
  • Legacy Adapter Vision Enable command line
    functionality to be brought in the the SOA
    service flow

Command Line Application
AdapterSrv
stream output
exec with arguments
-process()
Adapter Srv
Canonical XML Message Command Arguments
StreamHandler
ProgramOutput
start()
Response
Enterprise Service Bus HTTP, JMS, Virtual
Memory, File Endpoints
52
Legacy Adapters PatternAdapterSrv Receives XML
messages of commands
Adapter Pattern
AdapterSrv
Xml Msg Command Name Arguments
Xml Msg Command output
JIBX unmarshall XML
JIBX
Command Line Process
Enables Command Line Process to run as SOA Service
53
Legacy Adapters PatternuEngine Runs Command
Within Action Script
Adapter Pattern
Name arguments
AdapterSrv
JIBX marshall XML
Xml Msg Command
uEngine Can Run Command As If It Was Part of the
Script
54
Data Notification / subscription Enable
Automation of Products and Clients
Subscription Pattern
  • Notification / Subscription Vision Automate
    product generation and client product display
    based on new data ingested.
  • Implementation Any uEngine script can be made
    into a subscription based on ingest of new data
    or a time schedule.
  • New data triggers the execution of the cached
    uEngine script
  • The resulting product response is placed on a JMS
    Topic

55
CAVE Data Notification SubscriptionData
Matched to Menus by Optimized Decision Tree
Subscription Pattern
CAVE
DecisionTree
EDEX
CAVE Startup
Quinlan ID3 Algorithm
Data Item
AlertJobStarter
Build Data URI
Menu dataURIs
URICatalog
Message Received
AlertManagerJob
Data URI List
ingested dataURIs
quartz Timer
Data URI Aggregator
EDEX
Script Ids
topicedex.alerts
topicedex.subscription
Check scripts for URIs
56
Alerting Pattern Based on Ingest Warning GIS
located on Map and Displayed Tree
Subscription Pattern
ascii File//../data/sbn/ascii
ESB RegExFilter Fan Out Routing
Threaded Listener AlertManagerJob
warning jms//cp/tstormwarning
Locate Zooms In To Warning Area on Map
57
Repository Refinement
Subscription Pattern
  • Subscription added to PostgreSQL data base
  • Server state data simplifies clustering
  • Configurable Purge Retention Period
  • Performance enhancements with indexes on metadata

PostgreSQL Data Base
scripts
subscription
58
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

59
ADE Data Model IntroductionCanonical XML SOA
Interfaces Excluded
Data Model
  • Data Access Layer Implementation Using Hibernate
  • Data Access Object (DAO) Concept Leverages
    Hibernate
  • Data Persistence Through HDF5 Why?
  • High performance gaming level interactions
    supported
  • Chucking of data records supports visualization
    tiling
  • Flexible retrieval supports 4D rendering
  • Streaming compression
  • Meta Data Implemented in PostgisSQL through
    Inheritance
  • Defined only in plug ins, drives Data URI
  • Base Object Model is Extended in Plug Ins
  • Data URI Concept Ties Everything Together
  • Purging Concept of Circularly Repository
    Structures

60
ADE Conceptual Data Model DesignI/O Formats
Follows Existing Standards
Data Model
  • Output Formats
  • Vector ERSI shape file, SVG, Redbook
  • Raster GeoTIFF,png,jpg
  • Text Canonical XML, WMO Bulletins
  • VTEC warnings
  • Input Formats
  • grib1/2
  • GINI
  • ASCII (WMO, shef, )
  • Radar Level II

Data Flow
Internal Object Model
  • Request Formats
  • Canonical XML

T05 Data Access Layer
  • Decoded
  • Persistance
  • Respository
  • Flat Files
  • HDF5
  • RDBMS
  • Ingested
  • Persistance
  • Respository
  • Flat Files
  • HDF5
  • Metadata
  • Index
  • RDBMS
  • Raw Storage Model
  • Transmission Formats
  • HDF5 model
  • Decoded Storage Model
  • RDBMS Schema
  • HDF5 model
  • Static data model
  • MetaData Model
  • RDBMS Schema

61
Data Access Layer APIHibernate Leading Object To
Relational Approach
Data Model
  • Solves Fundamental Problem of Impedance Mismatch
  • Maps between Object Model and Relational Data
    Model
  • Provides Object based Query Facilities
  • Improves Performance over JDBC, Designed for
    Clustering
  • Reduces Code Count Improves Productivity
  • Built in Support in SPRING

Hibernate Enables Meta Data Performance and
Adaptability
62
Hibernate XML Object/ Relational MappingDefined
in SOA Plug In Enables Adaptability
Data Model
Satellite Plug In satellite.hbm.xml SatelliteRecor
d.java
Data Object Model
mapping
Data Object Model Extended By Plug In Follows
Base Model
Plug In Defined Object to Relational Mapping
by Hibernate XML
Plug In Enables Adapting To New Data Types
63
AWIPS-II Data Access PatternDesigned to
Support Clustering
Data Model
Mule ESB
Enables Thread Safe Access
  • DALconfig.xml
  • dataSource
  • hibernateProps
  • sessionFactory
  • DAOs

SPRING Dependency Injection Container
SPRINGs Hibernate Support
SessionFactory
ltltJavagtgt DataLayer
C3P0 JDBC Connection Pooling
ltltabstractgtgt BaseDao
saveMetadata() findMetadata() findPluginVersion
(plugin) executeQuery(query) executeUpdate(query
)
mappingJarLocations
persistMetadata() findMetadata() findPluginVers
ion(plugin) executeQuery(query) executeUpdate(qu
ery) generateTable(tableName)
plugins//.jar .hbm.xml
64
SOA Plug In Defines a Meta Data Table SetEach
Plug In Also Defines a HDF5 set
Data Model
RDBM Meta Data
HDF5 Repository
Circular Series
Circular Series
name1 Plug In
name1
Data URI Creates HDF5 Record Structure
nameN
nameN Plug In
Dynamic Meta Data Schema Follows Rules
Flexible Data Model is Plug In Extendable To New
Data Types
65
Plug In Creates New MetaData in RDBMSUses
PostgreSQL Table Inheritance and Rules
Data Model
Plug In
Auto generates Meta Data Schema At Plug In Load
Time
.db.xml
Insert Rule Chooses Sub Table As Function of Time
Name tableName Version hibClass
Circular Series of Sub Tables Enables A Self
Maintaining Schema
Sub Tables
66
Meta Data DemoUsing CAVEs Volume Browser
Data Model
Dynamically Populates Select Boxes Through
Catalog Queries
SOA Service ProductSrv
uEngine
Canonical XML Message Query Meta Data For Catalog
T05 All New
Hibernate
MetaData Store
67
Refines DataURI ConceptKey for System
Adaptability to New Data Types
Data Model
  • DataURI is a reference to data in the data store
    (i.e. D2D Data KEY)
  • Enables Automatic Subscriptions For all Ingested
    Data
  • Automatically ties data persistence to meta data
  • Enables Plug In Extendibility to new data types
    with changing any base code
  • T05 implemented a design for automatic generation
    of DataURIs

68
Meta Data Model Drives DataURIAuto generated
DataURI couples HDF5 to MetaData
Data Model
Auto Generates Data URI references from XML
Definition And Meta Decode of Ingested Record
Satellite Plug In satellite.db.xml SatelliteWriter
.java
IngestSrv
ltltJavagtgt SatelliteWriter
Example Data URI From a LOG File
HDF5 Repository
XML Meta Data Definition File Tag Specifies If
Element is Part of Data URI
RDBM Meta Data
Plug In Enables Adapting To New Data Types
69
Data Persistence Using HDF5HDF5 Files In Time
Ordered Bins Like MetaData
Data Model
Circular Time Bins
/awips/opt/data/hdf5
i.e. Autogenerated dataURI ties Metadata to HDF5
Record
Plug In 1
Plug In 2
Plug In 3
70
Data Persistence Using HDF5Application Code
Interfaces Through API
Data Model
Plug In Code Using HDF5 api
ltltinterfacegtgt IDataStore
ltltinterfacegtgt IDataRecord
addDataRecord() store() retrieve()
getsetDimension() getsetName() getsetSizes()
getDataObject()
ltltJavagtgt SatelliteWriter
write(record) dataStore (HDF5DataStore) Data
StoreFactory
ltltJavagtgt HDF5DataStore
ltltabstractgtgt AbstractDataRecord
addDataRecord() store() retrieve() -writeHDF(gr
oup, record) -createGroup() -unlockFile()
getsetDimension() getsetName() getsetSizes()
getDataObject()
jhdf5.jar
71
uEngine Using the Data Access LayerSingle API
enables uEngine to access all data
Data Model
uEngine
Results dataLayer.findMetaData (plugin,
fields, operands, values, sort,
count) createMetaDataMap()
HDF5 Repository
Dynamically Creates URI Reference to Data Record
72
Data Repository MaintanceBased on a circular
cycle of tables HDF5 files
Data Model
PurgeSrv
1 hr Wake Up
DbManager
Table holds retention period /2 of data (I.e.
12hr of data for 24 hr of retention time)
HDF5 directory tree
Plugin Metadata
day hour
30
1
delete
Metadata_1
Metadata_2
2
Metadata_3
Delete Referenced HDF5 files
new
Metadata_4
Purge creates new table and deletes oldest
73
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

74
AWIPS-II Visualization Overview (CAVE)Built
from Eclipse RCP Plugins is Extensible
Visualization
Eclipse RCP 3.3 XML User Localized Menus Tool
bar
GIS Display All data layers map projected,
scriptable, and subscribable
High Performance Image Rendering GPU Shader
Language Controls of (Projection,Color, )
QUAD Tiling of Raster Data World Wide Google
style zoom/pan
Data Sampling/Analysis Through raw data load
GIS Whiteboard Collaboration Projection
independent whiteboard, VoIP, Chat
Animation of All Data All displayed data
automatically under subscription
High Performance Vectors Through Vertex Array
Interface
Thick/Thin Client Through ESB Endpoint
Enterprise Service Bus Data Connections
Advanced GIS Visualization Enables Gaming
Interactions
75
CAVE EnhancementsMost Significant Update Since
Project Began
Visualization
  • Eclipse RCP 3.3 based, GeoTools 2.4, JOGL 1.1,
    Velocity, and batik
  • Entirely new raster rendering implementation
    based on raw float data
  • New raster tiling concept using HDF5
  • New color map rendering concept
  • New GPU mechanism for map reprojection
  • Implemented Display Bundles
  • JIBX extension for Resource to Bundle mapping
    enables quick adaption to new visualization
    resource
  • High Performance radar rendering similar to D2D
    all-tilts
  • On demand rendering
  • Active raster pixel data integration
  • Warning generation with Velocity parsing of
    templates

76
AWIPS-II Example Data Fusion in CAVE GIS
display of raster, point, and vector data
Visualization
Weather Satellite QUAD tiled display raster data
Lighting Strikes High frequency point data
converted to vector display
Data Spatially and Temporally Correlates
Point Weather Observations Text base observation
converted to vector display
Geo Political Maps Shape File rendering for
ERSI Map data
Complete Layer Controls Order, Color,
transparency, visibility, Active Layer
Automatic Subscriptions All displayed data
automatically under subscription and animates
Automatic Data Fusion for All Ingested Data
77
AWIPS-II Leverages Eclipse PerspectivesEnables
Multiple User Interfaces
Visualization
Weather Workstation
Hydrology Workstation
Graphical Forecast Editing
Python Editor
ESB
Localization
UtilitySRV
XML Driven Menus Color Tables Scripts
Config Maps
  • Perspectives leverages entire set of CAVE
    eclipse plug ins
  • Each perspective can have unique menus,
    toolbars, and dialogs
  • User can actively switch back and forth between
    perspectives

78
CAVE Menu Extensibility Eclipse
3.3Configuration Preferences, Menus, Data
Visualization
plugin.xml
Menus defined in plug in xml
CAVE satellite plug In
bundle.xml
references bundle
Menu fields update Automatically through the
AlertManagerJob as data is ingested
Embedded uEngine Script Retreives and transforms
data for display
79
CAVE Menu Extensibility PatternLeverages
Capability in Eclipse 3.3
Visualization
CAVE satellite Plug In
plugin.xml
Bundle IRWindow.xml
uEngine Script to Retrieve loop of data
80
Workstation Menus / Toolbars / RenderingXML
Configurable Menus XML Bundles
Visualization
.ui.personalitles.awips plugin.xml
.site. (from localization) plugin.xml
XML Display Resource Bundles
Commands
19 Map
plugin.xml Commands Parameters Menu labels
setScale
11 Plots
30 Display Resources
RadarDisplayControls
1 Skew-T
menuRetrieval
productURI productName variableList
8 Radar
11 Models
81
CAVE Display BundlesEnables Adding Procedures
and Display History
Visualization
XML Document Bundle Definition layers color
maps line widths
82
CAVE Visualization Service EndpointEnables
Gaming Style Data Interactions
Visualization
Eclipse RCP 3.2 Plug In Extendable Plug In for
Warn Generation Added
Radar Rendering Uses Dynamic raster tiling
GPU Shader Language Rendering Controls (Color,
), Animation
Added Bundles with Save / Retrieve
New Quad tiling of large raster Sets leverages
HDF5 chunking
Active Raster Data Interrogation
Dynamic map reprojection using GeoTools
Transforms GPU Warping
All Tilts Keyboard Controls
83
CAVE Radar RenderingRadial Data Gets Tiled
Dynamically
Visualization
ltltinterfacesgtgt
ICAVEResource IColormappableResource IImagingResou
rce ITimeSeqResource IVertSeqResource IToggleableR
esource IInspectableResource IBlendableResource
HDF5 Radar Data
84
CAVE Raster Data InterrogationAll raster and
shape files are inspectable
Visualization
  • Mouse X,Y Events are Coordinate Converted to
    Lat/Lon
  • The Lat/Lon location is transformed to retrieve
    the data out of the correct displayed tile at the
    x, y location
  • The data is transformed into displayable units

ltltjavagtgt RadarTypeRecord
dataURI RadialContainer AbstractTileSet Tim
estamp typeMap tiltMap
85
Workstation Plot Model and Maintenance Plot
Vector Library Converted to SVG (XML)
Visualization
Metars
Obs MetaData Parsed data fields
Plug In
Job
Buoys
PlotModel Generator
PlotResource
Progressive Disclosure
SVG Plot Model
SVG Plot Model
SVG Plot Models
UA plot 925MB
Entire present Wx symbol set converted to SVG
86
Workstation Warning Generation
Visualization
T09 Delivered
T010 Delivered
T08 Delivered
WarnGen GUI
Life Cycle Rules
Dissemination
VTEC Database
tornado.vm
severethunderstorm.vm
Templates 3..19
TextDB
GIS Capabilities Warning by PolyGon
UPDATE LIST
Severe Weather Statement
Store
Set VTEC
Generate Text
Send to TextWS
Text Workstation Editor
Restart
87
Workstation Text WS and Wx EditorImplemented as
2 RCP plug ins in CAVE
Visualization
.texteditor
.textworkstation
EDEX
TextDB
Workstation Interacts with TextDB
text Plug In
Hibernate Access Objects tablenameDao
../opt/data/sbn/test
Text Ingest Endpoint
88
GIS Map Reprojection By Texture WarpingGPU
Program 50 times Faster than CPU
Visualization
Analysis/Sampling enabled by the use of raw data
CAVE Visualization
HDF5 Repository
Decoded Blobs Grids Satellite Rasters Radar
Lighting
Chunked Float Array
Mapping Vertex Array
GIS Fused Layer
GPU Program
Color Map To 1D Graphic Texture
XML Color Tables RGBA Quads Unlimited
OpenGL gt 2.0 Graphical Processing Unit (GPU)
OpenGL API JOGL
Warp
CPU-Card Boundary
Lambert conformal Color Map
Enables Gaming Style GIS Data Fusion
89
GPU Fragment Shader Color MappingEnables
Interactive Color Controls
Visualization
ltltJAVAgtgt GLTarget
UsersRGB color map (RGBAs in XML format)
Grib Data Decoded to float array
texture1D RGB color
texture2D float Image data
index
gl_FragColor
Final Image
Interactive Color Maps 20 to 50 Times Faster Than
CPU
90
GIS Collaboration Capability Two LaptopsBased
on XMPP protocol standards
Visualization
AWIPS Laptop 1
AWIPS Laptop 2
Connecting to a Specific Collaboration
wildfire Collaboration
Wide Area Network
xmpp
Jabber Standard Collaboration Protocol
CAVE Menu Interface To Collaboration
Concurrent CAVE Perspective has Chat Area
Supports VoIP through plugin
91
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

92
AWIPS-II Server Execution StructureMarrying
ESB with Dependency Injection
Server Execution
  • ESBs evolved out of the expense of getting
    services to communicate across multiple protocols
    and languages
  • Ross Mason identified these problems back around
    2001 and developed the Mule Project which now has
    a large and active community
  • A parallel development by Progress Software
    created Sonic ESB and they coined the term
  • Mule ESB is based on Enterprise Integration
    Patterns and is the most elegant extensible
    solution
  • The high decoupling of Springs Dependency
    Injection Container with Mules endpoint abstract
    creates a high value approach
  • Solving Integration Problems Using Patterns
  • Messaging Systems, Channels
  • Message Construction
  • Message Routing
  • Message Transformation
  • Messaging Endpoints
  • System Management

Author Hohpe, Gregor Author Woolf, Bobby
Mule ESB Spring Becomes a Discriminator
93
Mule ESB SPRING ContainerServices decoupled
from transport mechanisms
Server Execution
MULE ESB
SEDA Load Balanced Queue
Mule End Point Provider
Inbound Router
Data
Data
Message Receiver
Connector
Transformers
Mule Interceptor Pattern
Logger
Mule End Point Provider
Outbound Router
Profiler
Message Receiver
Security Check
Transformers
Connector
EDEX SOA Service
EDEX SOA Service
  • JMS
  • Files
  • HTTP
  • FTP
  • E-Mail
  • EDEX SOA services are Dependency Injected
    Components
  • Configured in the XML Mule files

Data
Data
94
AWIPS-II Data Ingest EndpointsExample Warnings
Routed to Their Own Endpoint
Server Execution
SBN Satellite File//../data/sbn/sat jms//cp/sat
IngestSrv
pluginType SATELLITE
IngestSrv
IngestSrv
pluginType GRIB
Grib File//../data/sbn/grib jms//cp/grib
pluginType RADAR
pluginType MESOWEST
Radar File//../data/sbn/radar jms//cp/radar
mesowest jms//cp/mesowest
SEDA Scaled SOA Service
warning jms//cp/tstormwarning
ascii File//../data/sbn/ascii
metar jms//cp/metar
shef jms//cp/shef
ESB RegExFilter Fan Out Routing
alert jms//cp/alert
95
SOA Capability Available as ServicesScales Up
with SEDA Clustering on two Levels
Server Execution
  • Within Container
  • Within Data Server Cluster

SBN
CP.1
SBN Srv
JMS Broker
Automated Client
JMS//cp
Enterprise Service Bus
Data Server.1
Data Server.2
JMS//productSrv
.N
ESB Container
ESB Container
ProductSrv.1
IngestSrv.1
ProductSrv.1
IngestSrv.1
ProductSrv.2
ProductSrv.2
IngestSrv.2
IngestSrv.2
ProductSrv.n
ProductSrv.n
State
State
96
Server DeploymentsClustered JMS Broker Enables
SEDA load balancing
Server Execution
Deployment
Deployment
Data Server.1
Data Server.2
activemq_clustered
activemq_clustered
postgisSQL Server
ESB Container
ESB Container
JDBC Driver
HDF5 Lock State
HDF5 Lock State
HDF5 Files
uEngine
processing
postgisSQL Tables
97
Server State Synchronization of LocksEnsures
Data Integrity of HDF5 Writes
Server Execution
Local synchronized method for thread level
locking within JVM
Data Server.1
ESB Container
ltltJavagtgt HDF5DataStore
ltltJavagtgt ClusteredLockManager
HDF5 Lock State
store() -writeHDF(group, record) -unlockFile()
-localSynchronizationMap getLock(Lockname) relea
seLock(Lockname)
Multicast Voting Algorithm
JGroups
Clustered synchronized method uses a multicast
voting algorithm
From JBOSS J2EE Clustering
HDF5 Files
File Level HDF5 Locking For Cluster
98
Remote Management Through JMXGives Detailed
Insight Into Remote Java JVMs
Server Execution
Local Area Network (LAN) ie Demo is using a
Cross Over Cable
Server 1 CAVE
Server 2 Running JMX Console Remotely Monitoring
Server 1
activeMQ
Mule ESB SOA Services
99
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

100
Hardening JPROBE dynamic code analysis
Hardening
Jprobe Execution Time Breakout For Grib Ingest
Satellite Ingest
108ms
46ms/msg
64ms/msg
  • CAVE memory leak when running loops overnight,
    fixed
  • activeMQ dropping Mule connections (time outs),
    fixed
  • Jgroups configuration for clustering when
    clustering Con-ops determined
  • Message reference memory leak in AWIPS services
    fixed
  • Message file caching for idle messages eviction
    policy added to prevent memory growth

101
Omaha Test Bed ConfigurationDevelopment and Test
Hardware Resources
Hardening
Omaha Raytheon Server Room
CP Processor Dell 2550
Raytheon Firewall
awipscm Dell Server
Unidata LDM-6.6.5
LDM-6.6.5
Auto Nightly
2 Week Stability Testing
awips-dev1 2950 Server
awips-int2 2950 Server
awips-int1 2950 Server
Fiber Line
LDM-6.6.5
LDM-6.6.5
filters
filters
EDEX Cluster
EDEX Nightly
EDEX BldXX
EDEX BldXX
DVB-S
Orion Network
awips-db 2950 Server
awips-nas 2950 Server
postgreSql
Shared NFS
102
AWIPS-II Cumulative Memory ProfilesJMX memory
and cpu monitoring YourKit
Hardening
JMX cpu Useage 5.0 Day Average
YourKit Java Profiler
103
NotifySrv ChangesScripts and Subscriptions are
in RDBMS
Hardening
ltltSOA Servicegtgt NotifySrv
ltltSOA Servicegtgt AutoBldSrv
Script Ids
getDataURI() from New Ingested Data
getModifiedScript (subscribeKey, scriptID,
dataURI)
Run Script
getSubscriptionKey(dataURI)
Build Response
getSubscription ()
JMS Topic Subscription
Return Output
PostgreSQL Data Base
Send list of script Ids that Depend on the
Ingested Data
subscription
Compiled JavaScript uEngine script
CAVE
scripts
104
AWIPS-II Architecture Topics
Architecture
  • Architecture Introduction
  • Architecture Overview
  • Architecture of SOA Service
  • Architecture Patterns
  • Plug In pattern
  • Build pattern, Localization Pattern
  • uEngine pattern, GIS capability, Adapter pattern
  • Notification/Subscription pattern
  • Data Model
  • Visualization
  • Server Execution
  • Hardening
  • Security

105
Security Architecture ConceptsDo not change
anything significantly
Security
  • Each installation has perimeter security similar
    to what exists now we are not changing the
    security model.
  • Government is responsible for CA package
  • Goal is to be able to use a remote service as if
    it is local
  • Access Authentication
  • Service Authorization
  • Data Ingest Through Data Diode Type concept
  • SOA Endpoint Security Through ESB JMS provider

106
Proxy Gateway Enforces SOA PolicesEncapsulates
Customer Connection Polices
Security
Commercial Packages - JaxView, AmberPoint
SOA PROXY
  • Service Mediation
  • Protocol Conversions
  • Throttling
  • Load Balancing
  • Service Level Agreements

Authentication WS Security
En/Decrypt Message Content
Registry jUDDI
Blocking Non-compliant Requests
Opening Up The Enterprise
107
Security Concepts ContinuedSOA infrastructure
has built in capabilities
Security
  • JMS provider activeMQ
  • Support JAAS as a provider for authentication
  • Default points to a login.config on classpath
    that in turn points to
  • user.properties has userpassword info
  • group.properties has groupuser1,user2
  • User Roles on Queues and Topics
  • Read, Write, Admin (create)
  • Configurable message authorization policy allows
    each message to be content based authorized by a
    custom policy
  • Mule Security Acegi JAAS PGP
  • Built in configurable security manager is
    responsible for authenticating requests
  • Configurable encryption strategy includes
    transports such as SSL and HTTPS
  • JAAS security provider (Java Authentication
    Authorization Service) API in since Java 1.4
  • Mule is extendable to PGP signed encryption on
    endpoints for end-to-end communications

108
Security ArchitectureData Ingest Concept of Data
Diode for Firewall
Security
Low Side LDAD
High Side DX Cluster
Firewall
Initiates All Request And Data Transfers
Has No Clue About Other Side
Custom Socket Listener
Custom Socket Client
/clean-inbound/
/inbound/
One Way Interface
ESB Endpoint
Security Filter
HTTPS
ESB Endpoint
About PowerShow.com