Evolution of NPS and SPA METOC Metrics Work - PowerPoint PPT Presentation

1 / 77
About This Presentation
Title:

Evolution of NPS and SPA METOC Metrics Work

Description:

METOC Metrics: Design, Development, and Implementation Prof. Tom Murphree and CDR Rebecca Stone Naval Postgraduate School (NPS) murphree_at_nps.edu – PowerPoint PPT presentation

Number of Views:142
Avg rating:3.0/5.0
Slides: 78
Provided by: murp71
Category:
Tags: metoc | nps | spa | evolution | iraqi | metrics | navy | work

less

Transcript and Presenter's Notes

Title: Evolution of NPS and SPA METOC Metrics Work


1
(No Transcript)
2
(No Transcript)
3
Evolution of NPS and SPA METOC Metrics Work
Study Approach Selected Major Results
Cantu 2001 Analysis of NAVSAFECEN Class A aircraft mishaps Losses per year from weather-related mishaps 11 deaths 74M in damages
Martin 2002 Analysis of NAVSAFECEN Class A-B-C afloat mishaps Losses per year from METOC-related mishaps 2 deaths 900 lost/light duty days.
Hinz 2004 Analysis of forecasts, verifying observations, and mission planning and performance data from OIF CAOC Forecast POD 55. Negative ops impacts from mis-timing of late Mar 03 dust storm.
Anderson 2004 Analysis of model output, forecasts, and verifying obs for major OIF dust storm, late Mar 03 Mesoscale model errors led to errors in dust storm forecasts.
Jarry 2005 Analysis of performance and operational impacts of forecasts for Air Mobility Command (AMC) No go forecast accuracy 66. Weather mitigated missions saved by mitigation 67.
Butler 2005 Development of online near real time system for forecast performance and ops impacts metrics for NPMOD Fallon / NSAWC System in operational use since summer 2005. Missions saved or potentially saved by forecasts 36
Darnell 2006 Adaptation of Butler (2005) system for Air Combat Command (ACC) No go forecast accuracy 60. Missions saved or potentially saved by forecasts 37.
Callahan 2006 Adaptation of Butler (2005) system for carriers and amphibs Siprnet system created. Red POD 40 (preliminary).
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
4
Evolution of NPS and SPA METOC Metrics Work
Study Approach Selected Major Results
SPA 2001-2006 15 METOC operational impact studies for STK, ASW, MCM, EXW, NSW, and overall campaign Based on OPNAV planning scenarios Designed to investigate operational impacts of - uncertainty regarding environmental conditions - tactical utility of timely and accurate METOC information Uncertainty regarding METOC conditions has great impact on high-level mission success metrics. Mission planners require accurate information to predict performance and generate CONOPs. POM assessment community requires accurate information for CONOPs generation, force structure and allocation decisions, and requirements analysis.
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
5
(No Transcript)
6
  • Definitions
  • Metrics Objective, quantitative, data based
    measures of products and services. Examples
  • Metrics of product quality
  • Metrics of effects of products on customers
  • METOC Metrics Metrics of METOC organizations
    products and impacts. Two main types
  • Performance metrics metrics of capacity,
    readiness, quality, efficiency / return on
    Investment
  • Impacts metrics impacts on warfare customer
    operations (planning, execution, post-assessment)
  • Methods for Generating METOC Metrics
  • ? Collect / analyze real world data on METOC and
    customer ops
  • ? Model METOC and customer ops

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
7
  • Goals
  • 1. Develop and transition to operational use
    systems for
  • a. collecting data from METOC units and
    their customers
  • b. quantifying METOC performance and
    impacts on customer
  • operations
  • c. modeling and predicting impacts of METOC
    support on
  • war fighting operations
  • 2. Identify methods for improving quality and
    efficiency of METOC support.
  • 3. Recommend
  • a. focus directions for METOC resources
  • b. methods to incorporate METOC into OPNAV
    assessments

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
8
Effects of Environmental Uncertainty on Mission
Planning and Execution
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
Accurate Predictions
  • Opportunity Costs
  • Actual is better than predicted
  • Over-prepared for environmental risks
  • Spend too many resources, too much time
  • Resources better used for other locations/missions
  • Effectiveness Costs
  • Actual is worse than predicted
  • Under-prepared for environmental risks
  • Deploy too few resources, too little time
  • Increased risk of mission failure

Need to know relationships between forecasts,
planning, and outcomes in order to quantify, and
eventually reduce, outcome uncertainty and
associated costs.
9
  • Develop automated system to collect and
    analyze real world
  • data.
  • Develop and apply warfare mission model.
  • Use real world data to modify and verify
    model.
  • Use model to identify additional data to
    collect and analyses to
  • conduct.

10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Forecast Performance and Operational Impacts
Metrics OIF
From NPS thesis research of LCDR Jake Hinz, USN,
2004
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
14
(No Transcript)
15
Forecast Performance Metrics AMC
Overall performance very good. But high overall
FAC comes mainly from relatively common Go
events. Persistent Go forecasts would have given
almost the same overall FAC. FAC and skill for
relatively uncommon No Go events much lower. In
No Go cases of critical importance to operators,
forecast performance much lower than overall.
From NPS thesis research of Capt Jeff Jarry,
USAF, 2005
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
16
Operational Impacts Metrics AMC
Saved Mission a. No Go forecast for original
plan is accurate and b. Go forecast for
accepted mitigation plan is accurate and c.
Mission successful using mitigation plan
and d. Mission would have failed using
original plan.
From NPS thesis research of Capt Jeff Jarry,
USAF, 2005
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
Operational Impacts Metrics Responses to
Forecasts of Negative Impacts ACC PACAF
  • Percent of missions for which
  • Planning forecasts indicated negative weather
    impacts 36
  • Planning forecasts led to mission plan changes
    21
  • Execution forecasts indicated negative weather
    impacts 39
  • Execution forecasts led to mission plan changes
    21
  • Missions that experienced negative weather
    impacts 36

Missions that avoided, or could have avoided,
negative weather impacts based on forecasts
issued ? 36 for all our real world Air Force
and Navy data sets.
From NPS thesis research of Maj Karen Darnell,
USAF, 2006
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
31
  • Online Real World Metrics System Users
  • In operational use by Naval Pacific Meteorology
    and Oceanography Detachment at NAS Fallon in
    support of Naval Strike and Air Warfare Center
    (NSAWC).
  • System in operational testing by Enterprise,
    Eisenhower, Stennis, and Iwo Jima.
  • Web site and database components of system being
    transitioned from NPS to FNMOC. Beta version
    expected on FNMOC development server winter 2007.
  • System adapted and tested for use by USAF Air
    Combat Command (ACC) and Pacific Air Forces
    (PACAF).
  • Adaptations of system for other warfare areas
  • System being adapted for use in NSW
    (collaboration with NOSWC)
  • Discussions underway to adapt system for
    automation of ocean model product metrics
    (collaboration with NRL-SSC)
  • Planning underway to adapt system for use in ASW
    (collaboration with ASW directorate)

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
32
See brief by CDR Steve Woll for more on uses of
metrics by METOC leadership.
33
See brief by CDR Steve Woll for more on uses of
metrics by METOC leadership. See brief by CDR
Mike Angove for more on uses of metrics in
assessment / budget processes.
34
Integration of Real World and Model Metrics
See briefs by Luke Piepkorn Matt McNamara for
more on our metrics modeling efforts.
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
35
  • Future Extensions - Approach to Other Warfare
    Areas
  • Develop metrics dashboard for all directorates
  • STW work directly applicable to other
    mission-centric warfare areas
  • NSW, ISR, EXW / AMW
  • Outcomes measure performance and impacts (e.g.,
    missions saved), improve mitigation
    recommendations, improve CONOPS
  • Applicable with modification to campaign style
    warfare areas
  • ASW, MIW
  • Model metrics with fewer real-world exercises
    and operations to analyze, real-time realistic
    modeling provides opportunity to explore options,
    what-ifs
  • Well-positioned if/as JFMCC moves to Maritime
    Tasking Order concept
  • In all warfare areas, need to
  • Go from anecdotal, often post-mission assessments
    to objective, automated, real-time, assessments
    that can be incorporated into planning process at
    strategic, operational, and tactical levels
  • Inject realistic METOC factors and forecasting
    into Naval modeling, simulation, and assessments

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
36
Future Extensions - ASW
Metrics as tools for evaluating different METOC
support strategies
From ASW Coordination/CONOPs Conf 14 Mar 05 CAPT
Jeff Best, CNMOC Director for ASW CDR Van
Gurley, CNMOC Deputy Director for ASW
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
37
Future Extensions - ASW
Metrics as tools for evaluating different METOC
support strategies
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
38
Future Extensions - ASW
Metrics as tools for evaluating different METOC
support strategies
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
39
Future Extensions - ASW
  • Preliminary Proposal
  • METOC metrics program for ASW should be designed
    to
  • Develop ASW real world data collection, analysis,
    and reporting system
  • Develop ability to model ASW operations and METOC
    support of them
  • Use real world and model systems to test and
    experiment (e.g., with variations in objectives,
    weighting of objectives, means to meeting
    objectives, success measures, etc.)
  • Report results rapidly, automatically, and in
    optimal formats to both METOC personnel and ASW
    customers
  • Support facts-based decision making at multiple
    levels by both METOC personnel and ASW customers

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
Backup Slides
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
44
Dollars and Metrics
  • Metrics are Quantitative Measures of Performance
  • May relate performance to other areas
  • Visibility forecast accuracy and CVN green deck
    for air ops
  • Performance to Customer Dollars
  • Iterative Process
  • Do not need to be dollar related initially!

From NPS thesis research by LCDR Jake Hinz, 2004
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
From NPS thesis research of Capt Jeff Jarry,
USAF, 2005
For official use only
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
WIAT Simulation of Adaptive Strike
MissionPlanning Process
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
57
WIAT METOC Features
  • Includes forecasted and actual conditions over
    time from OIF weather briefing maps
  • Spatial resolution 25 x 25 nm
  • Temporal resolution varying, down to 2 hrs
  • Represents forecasted and observed cloud density,
    cloud layers
  • Includes inputs for other forecasted and observed
    conditions
  • Simulates interactions of OA division and mission
    planners
  • Allows forecasts of various accuracies to capture
    operational effects of forecast uncertainty

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
58
WIAT TACAIR Features
  • Force Composition
  • Size Carrier Air Wing
  • Sorties modeled as TACAIR missions
  • 2 AC per mission
  • Type F-18
  • Objectives
  • Provide CAS / KI to ground forces in transit and
    during engagements
  • Engage fixed strike targets targets of
    opportunity
  • Behavior
  • Fly missions according to the adaptive ATO
  • Sample Metrics
  • Mission modifications due to weather
  • 24 hrs prior 1 hr prior
  • Sortie missions aborted / sorties successfully
    flown
  • Weapons correctly selected / dropped
  • TACAIR mission success (targets destroyed / KI
    stations maintained / CAS successes)
  • Opportunity costs (targets that were incorrectly
    avoided due to erroneous forecasts)

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
Future Extensions - Approach to Other Warfare
Areas
  • Determine METOC inputs for warfare area
  • Current support products
  • Mission planning timelines
  • Collect product and observational data
  • Historic database information
  • Current operational observations
  • Develop applications which quantify
  • Value of METOC support
  • Impacts of improved environmental analysis and
    prediction
  • Capitalize on this information to improve METOC
    support
  • New or improved support products
  • New methods of feature analysis

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
66
Future Extensions - ASW
  • ASW Metrics Operations Analysis Task Areas
  • Task 1 Develop realistic vignettes and
    scenarios for ASW.
  • Consistent with both MCOs and exercises /
    operations
  • High level metrics motivated by OPNAV
  • Task 2 Identify the environmental factors that
    influence ASW platform and system performance.
  • Identify impacts and prioritize
  • Leverage existing vetted assessments for OPNAV
    and METOC communities
  • Task 3 Describe and model the ASW mission
    planning process.
  • Identify decision points and structure the ASW
    decision process in terms of where the decision
    is being made, the nature of the decision and
    options available,
  • Identify the likely options selected as a result
    of alternative environmental intelligence, and
    the uncertainty associated with the METOC
    products provided.

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
67
Future Extensions - ASW
  • ASW Metrics Operations Analysis Task Areas
  • Task 4 Develop an engagement model for ASW.
  • Realistic and credible representation of forces,
    tactics, and outcomes
  • Output measures of effectiveness that are
    consistent with OPNAV N81/N84 high-level
    measures of ASW success and what-it-takes-to
    win criteria
  • Representation of the environmental context of
    the mission with particular attention to the
    critical environmental features (collaboration
    w/NRL-SSC)
  • Representation of the mission planning process
    and the ability to input and implement the
    metrics collected from operating forces by NPS
    for their portion of the collaboration

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
68
Future Extensions - ASW
  • ASW Metrics Operations Analysis Task Areas
  • Task 5 Perform sensitivity analysis on METOC
    product accuracy and timeliness.
  • Coordinate with NPS and NRL-SSC to develop a set
    of cases and sensitivities to assess the tactical
    utility of METOC support services and to identify
    areas where accuracy and timeliness should be
    improved
  • Perform sensitivities to scenario variables such
    as threat, U.S. forces, and rules of engagement
    as well as sensitivities to the accuracy of
    forecasts, timeliness of METOC support, season,
    and other environmental uncertainties. Modify
    models and databases as required as a result of
    these tests and sensitivities.
  • Task 6 Provide reports, briefs, and analysis
    results
  • Deliver modeling tools and datasets together with
    documentation and training as directed.

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
69
Future Extensions - NSW
OOD Effects on NSW Number of Missions Impacted
by Ocean
FID Missions Minimally Impacted
ISR Missions Strongly Impacted
Number of Missions
MIssion Type
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
70
Future Extensions ASW / MIW
OOD Effects on ASW / MIW Number of Contact
Hours Added by OOD
OOD shows higher than expected MDRs. Screen/sensor
spacing increased. Significant improvement in
Contact Hours.
OOD shows lower than expected MDRs. Screen/sensor
spacing reduced. Significant improvement in
Contact Hours.
MDRs
Contact Hours Added
OOD Historical Traditional Minimal
improvement in Contact Hours.
Day
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
71
  • Future Extensions - CNMOC Use of Metrics
  • METOC Metrics Dashboard During Realignment

ASW MIW NSW ISR FLT AVN MAR NAV PTA
Capacity
Readiness
Quality
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
72
  • Future Extensions - CNMOC Use of Metrics
  • METOC Metrics Dashboard After Realignment
    Completed

ASW MIW NSW ISR FLT AVN MAR NAV PTA
Capacity
Readiness
Quality
Impact
Efficiency
Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
73
Future Extensions - ASW
ASW Example
Mission Planning Steps
  • Determine length and position of barrier desired
    between threat ports and op areas
  • Determine the barrier performance objective
    (e.g., 0.90 P(detect) against quiet transitor)
  • Compute forces required based on environment,
    sensor performance, and threat/searcher speed
  • METOC Issues
  • What is the impact of accurate knowledge and
    modeling of bottom
  • properties on mission planning?
  • Is the age of the information important?
  • Are more bottom surveys necessary?

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
74
(No Transcript)
75
Future Extensions - ASW
Exercise Support Results
How did the actual search performance change as
a result of the improved environmental
information and subsequent CONOPs change?
The total actual performance improvement of 9
hours comes from two sources
  • Improvement due to a more favorable environment
    (4 hours)
  • Improvement due to understanding the environment
    and changing the CONOPs (5 hours)
  • Search performance does improve nearly as much
    as expected
  • Median time to detect both threats improves from
    23 hours to 14 hours
  • Probability of detecting both threats by the end
    of the exercise improves from 75 to 85

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
76
  • Future Extensions - NSW
  • Proposal Submitted to SPAWAR for FY07-FY09
    Project
  • Project Performers
  • NPS Tom Murphree, CDR Rebecca Stone
  • SPA Paul Vodola, Luke Piepkorn
  • NRL-SSC Rick Allard, Gregg Jacobs
  • CSI Bruce Ford
  • Project Objectives
  • Develop METOC metrics data collection methods,
    databases, data analysis, modeling, and metrics
    computation systems for METOC support of Naval
    special warfare (NSW) operations.
  • Apply systems to
  • collect and analyze data
  • simulate impacts of METOC products on warfighting
    operations,
  • determine metrics that quantify the performance
    and operational impacts of METOC forecasts
  • Transition to METOC community a metrics toolset
    composed of data collection, database, data
    analysis, and modeling systems for NSW.
  • Develop recommendations for METOC leadership.

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
77
  • CNMOC Roles
  • Advocacy at CNMOC
  • Systematic application of integrated real world
    and model based metrics methods to all
    directorates
  • N6 / EED staffing
  • Coordination with TD
  • Assistance in establishing
  • METOC metrics working group
  • METOC metrics advisory board
  • Assistance in establishing METOC metrics as an
    NPS thesis research priority
  • METOC metrics topics (meteorological and
    oceanographic) on future thesis topics lists
  • Guidance to NPS students (e.g., email)
  • Support for further funding

Murphree et al., METOC Metrics, Jan 07,
murphree_at_nps.edu
Write a Comment
User Comments (0)
About PowerShow.com