Cultural Influence on the Implementation of Lessons Learned in Safety and Risk Management Decisions

1
Cultural Influence on the Implementation of
Lessons Learned inSafety and Risk Management
Decisions Isaac Maya, Najmedin Meshkati,
Mansour RahimiDeepak Madabushi, Kevin Pope,
Meredith SchulteCPMR Fellows MeetingJanuary
19-20, 2005
N a t i o n a l S c i e n c e F o u n d a t i o
n E n g i n e e r i n g R e s e a r c h C e n
t e r
2
USC-NASA/CPMR PROJECT MANAGEMENT

• Project Team consists of PI and 2 Co-PIs with
  complementary skills, 2 Graduate Students, 1 UG
• Dr. Maya -- 26 years experience in engineering
  RD and executive, program and project management
• Dr. Meshkati -- experienced engineering
  researcher in reliability, human factors, safety
  culture, risk management of large-scale, complex
  technological systems NASA Faculty Fellow
  2003-04
• Dr. Rahimi -- experienced engineering researcher
  in system safety, human computer interaction
  (HCI) for advanced technology systems
• Three Excellent Students -- Two Masters students
  in Industrial Systems Engineering, undergrad in
  anthropology and biology
• Technical Advisory Board
• Industrial, academic, and NASA/JPL experience

3
Integrated Media Systems Center
NSF Engineering Research Centera partnership in
pursuit of research and innovation in multimedia
and immersive Internet technologies and their
applications 28 Investigators and 260 students
in partnership with National Science
Foundation University of Southern
California Viterbi School of Engineering
Ranked 6th in US, 135M/yr in CG
funding Annenberg Center for Communication Commer
cial Partners Computer Hardware and Software
Aircraft, Aerospace, Defense
Petroleum, Oil, Gas Telecommunications Entertai
nment Other Government Agencies DARPA, NASA,
JPL, NIMA, ONR, U.S. Army
4
IMSC Program Management Process

• Application requirements and relevance
• Collaboration and Integration
• Research
• Education and Outreach

5
IMSC Program Management Process for USC-NASA/CPMR
Project

• NASA/CPMR project requirements and relevance
• JPL collaboration and integration
• Project Research
• Education and Outreach

6
Research Highlights Applicable to NASA/CPMR
Project
IMSC has produced ground breaking results and
fundamental research applicable to NASA/CPMR
efforts

• immersive audio
• multichannel and HRTF approaches - holistic DSP
  approach
• information management and multimedia databases
• distributed and scalable data analysis and query
• computer vision
• computational framework for grouping based on
  tensor voting, tracking for augmented realities
  and SFX
• graphics animation
• 3D DSP mesh processing, compression, mesh
  operations, hair modeling and animation

7
Technical Advisory Board for NASA CPMR Project

• ACADEMIC
• Dr. Randolph Hall, Professor, Sr. Associate Dean
  of Research, Viterbi School of Engineering, and
  Director of USCs Homeland Security Center (Risk
  and Economic Analysis of Terrorism Events)
• Dr. Stan Settles, Professor, USC, member of the
  National Academy of Engineering
• NASA/JPL
• Dr. Ken Atkins (JPL, Retired), development
  project manager for the fourth mission in NASAs
  cost-capped Discovery program Stardust
• Dr. Steven Prusha (JPL)
• Dr. Payman Arabshahi, Senior Research Staff,
  Section 331 (Communication Systems and Research),
  JPL
• INDUSTRY
• Mark George, Sr. Manager, RD, Lockheed Martin
  Space Company
• Eric Honour, President, Honourcode, Inc., past
  President, International Council on System
  Engineering, and Director, Systems Engineering
  Center of Excellence
• Dr. George Friedman, Retired, Northrop, and
  Founder, past President and Fellow, International
  Council on System Engineering
• Dr. Milton F. Pope, Systems Engineering
  Directorate, Evolved Expendable Launch Vehicle
  Division, The Aerospace Corporation

8
Collaboration with NASA Personnel and Technical
Advisory Board (TAB)

• TAB Telecons
• Dr. Ken Atkins visits at USC and JPL
• Stephen Prusha, Manager, Strategic Systems
  Technology Program, JPL, information exchanges
• Dudley Killam, Operations Manager, Office of
  Safety and Mission Success, JPL, visits
• Carol Dumain, JPL, David Oberhettinger, NG, and
  JPL Lessons Learned Committee
• Al Gallo, Goddard information exchange
• Jeanne Holm, Chief Knowledge Architect, and
  Manson Yew, JPL visits
• Manson Yew and staff visit at JPL
• Mark George and LM personnel

9
Methodology and Sources

• Direct discussions with external collaborators
  (NASA/JPL and industry)
• Extensive literature review and analysis
• In-depth review of NASA Lessons Learned
  Information System (LLIS)
• Analysis of LLIS data entries
• Human Factors studies

10
Summary Findings

• Mishap investigation boards have identified
• Technical lessons learned
• Need for NASA culture change to reap benefits
• Our preliminary LLIS data analysis shows
• Culture change to implement LL processes has been
  evolutionary and Center-dependent
• LLIS use is not measurably tracked
• Found evidence that system-level management
  change can overcome LL culture change barriers
• Currently, several processes and protocols in use
• No NASA-wide, standard, system-level LL template
• JPL has successfully implemented comprehensive LL
  process addressing technical and culture change
  challenges

11
Summary Findings

• New NASA Engineering Network (NEN) is in final
  stages of beta testing before release
• Modernization of LLIS with current IT
• Addresses IT aspects of LL
• Limited address of system-level issues
• Human factors and systems perspectives and
  advanced information management technologies
  offer significant potential for
• Addressing culture change barriers
• Enhancing usability of NEN
• Increasing effectiveness of overall LL process

12
PRESENTATION OUTLINE

• Introduction
• Objective and Focus
• Literature Review Findings
• Cultural Issues Findings
• Lessons Learned Systems and NASA LLIS Findings
• NASA and JPL Management and System Processes for
  Lessons Learned

• NASA Engineering Network (NEN)
• Human Factors Studies
• Education and Outreach
• Summary Findings
• Conclusions and Recommendations

13
INTRODUCTION

• General impression that NASA Lessons Learned from
  past mishaps not being applied effectively in
  subsequent programs/projects
• Contributing factor could be that NASA culture
  features lead to this result
• Examine NASAs Lessons Learned Information System
  (LLIS) for data
• Compare to Lockheed Martin culture, lessons
  learned system and data

14
Objective and Focus of Research

• Research influence of culture on use of LL system
• Focus on three of the most frequently cited
  recurring causal factors for project failures
• Poor team communication
• Inadequate management review process in all
  life-cycle phases of the project
• Inadequate use of systems engineering (project
  management) principles for LL in complex/large
  scale projects
• Look for mishap, LL analysis and implementation
• Consult experts and examine LLIS for data

15
Selected Literature Review Findings

• Report to the Subcommittee on Space and
  Aeronautics, Committee on Science, House of
  Representatives (GAO-02-195) named Better
  Mechanisms needed for sharing Lessons learned,
  January 2002.
• http//www.gao.gov/new.items/d02195.pdf
• No assurance Lessons Learned being applied
• Unfamiliarity with lessons learned across
  centers/programs
• Cultural barriers inhibit sharing, capturing and
  submitting lessons
• Lack of support from agency leaders
• Success in industry comes from commitment to
  knowledge sharing

16
Selected Literature Review Findings (cont.)

• Earlier survey of NASAs Project/Program Managers
  on issues of lessons learned and the process
  (GAO-01-1015R) Survey of NASAs Lessons Learned
  Process, September 2001.
• http//www.gao.gov/new.items/d011015r.pdf
• Topics included collection and sharing of
  lessons, strengths and weaknesses of lessons
  learned processes and procedures, comments on
  LLIS, etc.

Source GAO-01-1015R, Survey of NASAs Lessons
Learned Process, September 2001.
17
Selected Literature Review Findings (cont.)

• Presentation at CPMR Workshop December 9-10, 2003
  by the Best Practices and Lessons Learned Working
  Group Report Panel
• Examined integration of best business practices
  and Lessons Learned
• Discusses road blocks and challenges in achieving
  such an effort
• Aha, D.W., Weber, R. (2000). Intelligent
  Lessons Learned Systems Papers from the AAAI
  Workshop, Menlo Park, CA AAAI Press Weber, R.,
  Aha, D.W., Becerra-Fernandez, I. (2001).
  Intelligent lessons learned systems.
  International Journal of Expert Systems Research
  Applications, Vol. 20, No. 1,
  17-34. http//www.kmlab.fiu.edu/Papers/Intelligent
  20Lessons20Learned20Systems.pdf
• Discussed implementation issues of lessons
  learned in any general organization.
• Comparisons can be made on aspects of current
  NASA system
• Could be used as a benchmark in the second phase.
• Characterizations
• Content, Role, Orientation, Duration,
  Organization Type, Architecture, Attributes and
  Formats, Confidentiality, Size

18
Selected Literature Review Findings (cont.)

• NASA Culture Survey, Assessment and Plan for
  Organizational Culture Change at NASA, March 15,
  2004.
• Three key themes emerge from the survey data
• Overall NASA has strong work-group level teamwork
  and communications.
• Overall NASA has improvement opportunities in
  upward communications about safety and in
  employee perceptions about the extent to which
  the organization cares about employees.
• Overall there is little variation among NASA
  locations, among offices within NASA locations,
  or between programs.
• Executive Summary of Survey
• NASA personnel are committed to safety, but NASA
  culture not fully supportive of safety Open
  communication not the norm, people not fully
  comfortable raising safety concerns to management
• People do not feel respected by organization
  Technical commitment does not translate into
  organizational commitment
• Excellence is treasured for technical but not for
  management, support, communications
• Integrity understood, but mixed signals from
  management about raising issues.

19
Initial (General) Application to Current Effort

• Communication Between Centers
• General inability to effectively collect and
  disseminate lessons learned
• PMs lack of awareness of lessons being learned in
  other centers
• Lessons learned mainly through group meetings,
  project reviews
• Project Review and Management Processes
• Importance and priority of collecting, sharing
  and use of the lessons as a key component of NASA
  management goals and objectives not clear
• Incentives, resources and appreciation associated
  with adding to/referring to lessons learned not
  present
• Perception that discussion of problems not
  welcome, or problems too easily dismissed,
  reduces LL contributions
• Implementation of Systems Engineering Concepts
• Redundant/exclusive/varied systems for sharing
  lessons within NASA despite existence of LLIS
• Different methods/processes/systems exist for
  review and approval of lessons learned for LLIS
• LL systems engineering design lacks incentives
  and feedback

20
Findings from Literature Review and Collaborative
Discussions

• Cultural
• Notion that lessons not apt, relevant or useful,
  lack of time for LL (input/use)
• Communication of LL among Centers and HQ is
  incomplete, leading to inconsistent sharing,
  learning and application of lessons
• Tools and System Processes
• Need better LL IT tools and project management
  review processes and check points
• Organizational
• Management-driven initiative and systems-based
  process are needed for effectively collecting and
  sharing lessons
• Value and importance of LL to organizational
  goals needs to be emphasized and put in practice
  at all centers

21
Overview Of Lessons Learned Systems

• Many organizations use LL systems
• Elaborate list of LLS worldwide at
  http//home.earthlink.net/dwaha/research/lessons.
  html
• Many examples
• FAA
• NASA Aviation Safety Report System
• Licensee Event Report (NRC)
• DoD (Army, Navy, etc.), DoE
• Contractors (LM, NG, etc.)

22
NASA LLIS Research Plan

• Familiarize ourselves with current LLIS database
  and historical evolution
• Early development in Code Q, limited capabilities
• Assignment to HQ/Chief Engineer, expanded use
• Goddard maintenance, full deployment (current)
• Knowledge management (future)
• Evaluate the timeline of lessons based on impact
  of mishaps, incident review boards and GAO
  reports
• Graphically review data for each Center to
  determine trends, and study variations in the
  approval process

23
Some Results
24
Some Encouraging Results
Progress in Reducing Approval Time
25
A Lessons Learned Success Story at JPL From Mars
Climate Orbiter and Mars Polar Lander
Mishaps/Failures in 1999 to Mars Explorer Rovers
Successes in 2003

• Major Noteworthy Actions/Outcomes
• Creation of the Mars Climate Orbiter (MCO) Mishap
  Investigation Board (MIB)
• MCO Phase I Report Released November 10, 1999
  (identified root causes and factors contributing
  to MCO failure)
• Report on Project Management in NASA by the MIB
  March 13, 2000 (Recommendations for a culture
  shift by focusing on four categories of themes,
  which, in addition to the Mars Program at JPL,
  were also applicable to other programs
  throughout NASA (p. 36).
• People, Process, Execution, and Technology

26
A Lessons Learned Success Story at JPL From Mars
Climate Orbiter and Mars Polar Lander
Mishaps/Failures in 1999 to Mars Explorer Rovers
Successes in 2003 (cont.)

• Major Noteworthy Actions/Outcomes (cont.)
• Culture change based on lessons learned via
  development of systematic new project and process
  management tools and practices
• Clarification of major responsibilities of
  project managers concerning project quality,
  schedule and budget and understanding their
  interactions and trade-offs

27
A Lessons Learned Success Story at JPL From Mars
Climate Orbiter and Mars Polar Lander
Mishaps/Failures in 1999 to Mars Explorer Rovers
Successes in 2003 (cont.)

• Major Noteworthy Actions/Outcomes (cont.)
• Codifying and standardizing practices, and
  development of procedures
• Corrective Action Notices
• Design, Verification/Validation and Operations
  Principles for Flight Systems (D-17868)
• Development of the Flight Project Life Cycle
  concept and its enforcement of its multiple
  reviews
• Concept Review, Preliminary Mission System
  Review Project/System Preliminary Design Review
  Project/System Critical Design Review, Assembly,
  Test, and Launch Operations Readiness Review
  Mission Readiness Review Post Launch Assessment
  Review and Critical Events Readiness Review
• Flight Project Practices (Revision 5, 2/27/03)

28
A Lessons Learned Success Story at JPL From Mars
Climate Orbiter and Mars Polar Lander
Mishaps/Failures in 1999 to Mars Explorer Rovers
Successes in 2003 (cont.)

• Major Noteworthy Actions/Outcomes (cont.)
• Over 100 Return-to-Flight criteria and
  lessons-learned developed as a result of review
  boards were implemented in Rover program

29
JPL Lessons Learned Committee Manages Process and
Input to LLIS
30
JSC and Ames LL Process Flow Charts
31
NASA MANAGEMENT AND SYSTEMS PROCESSES FOR LESSONS
LEARNED
32
NASA ENGINEERING NETWORK PROTOTYPE STUDY
Learning Process Occurs Behind All
Components Embed lessons into tools and
communities

• New capability
• New portal
• Push vs. pull
• Improved search
• Metadata/mining
• Collaborative tool
• Query experts

Interagency/Aerospace Lessons Learned
Center Lessons Learned
NASA Lessons Learned
Collaborative Tools (NX, Jabber, WebEx)
Exploration Systems Project Environment
Metasearch
Training APPL SOLAR NET
Policies and Procedures
Feedback
Advanced Engineering Tools
Feedback
Responsibility Areas Chief EningeerGreen
Agency ResourcesBlue
Document and Data Repositories
33
HUMAN FACTORS (HF) IN LESSONS LEARNED SYSTEM
DESIGN

• HF-based design of HCI systems such as LLIS/NEN
  can significantly improve user interaction and
  efficiency
• Our preliminary usability testing results of
  HF-based redesign of LLIS interface indicate
  tasks such as
• Conducting searches
• Browsing search results
• Refining a search
• Can
• Be carried out in significantly less time
• Reduce potential for confusion

34
HUMAN FACTORS (HF) IN LESSONS LEARNED SYSTEM
DESIGN (cont.)

• HF methods can apply to future NEN design
  iterations, and can yield significant benefits
• Interface design
• Navigation efficiency
• Searching effectiveness
• Usability testing-based design of NEN
• Cognitive compatibility of NEN user with system
  design

35
EDUCATION PUBLIC OUTREACH

• Lessons learned and results of this effort
  integrated in
• Graduate-level PM class (ISE 515, taught by Dr.
  Maya)
• Two courses in engineering work design and human
  factors engineering (ISE 370 and ISE 570, taught
  by Dr. Meshkati and Dr. Rahimi)
• Educational lessons learned, case studies and
  classroom experience will be distributed via
  APPL, PMI, NASA, INCOSE and other industry and
  university channels
• Short courses integrating LL for PMs in NASA and
  aerospace industry, such as those offered by the
  USC Aviation Safety (and Security) Program, will
  be developed

36
Summary Findings

• Mishap investigation boards have identified
• Technical lessons learned
• Need for NASA culture change to reap benefits
• Our preliminary LLIS data analysis shows
• Culture change to implement LL processes has been
  evolutionary and Center-dependent
• LLIS use is not measurably tracked
• Found evidence that system-level management
  change can overcome LL culture change barriers
• Currently, several processes and protocols in use
• No NASA-wide, standard, system-level LL template
• JPL has successfully implemented comprehensive LL
  process addressing technical and culture change
  challenges

37
Summary Findings

• New NASA knowledge management system, NEN, is in
  final stages of beta testing before release
• Modernization of LLIS with current IT
• Addresses IT aspects of LL
• Limited address of system-level issues
• Human factors perspectives and advanced
  information management technologies offer
  significant potential for
• Addressing culture change barriers
• Enhancing usability of NEN
• Increasing effectiveness of overall LL process

38
Conclusions and Recommendations

• NASA LL culture change is possible and underway
• Executive level systems-based LL guidance and
  support critical for NASA-wide implementation
• Systems-based incentives, check points,
  consistency
• Successful JPL process can be adapted to address
  needs of other Centers
• NEN will provide much needed upgrade of LLIS
• Can effect culture change as part of
  systems-based approach
• HF-based HCI design can significantly enhance
  usability, acceptance effectiveness
• Advanced IT features can address major challenges