IT Security Metrics - PowerPoint PPT Presentation


PPT – IT Security Metrics PowerPoint presentation | free to download - id: 1eb4ae-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

IT Security Metrics


Goal: To train Federal IT security personnel how to develop metrics that they ... After completing this workshop, ... a password cracker that is. run regularly. ... – PowerPoint PPT presentation

Number of Views:222
Avg rating:3.0/5.0
Slides: 39
Provided by: sharonk150
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: IT Security Metrics

IT Security Metrics A Practical Approach to
Measuring Information Security Measuring
Security at the System Level
IT Security Metrics Training
  • Audience Federal IT security personnel with
    GISRA reporting responsibilities
  • Goal To train Federal IT security personnel how
    to develop metrics that they can use immediately
    to assist with GISRA reporting
  • Duration 3 hours

  • After completing this workshop, you will be able
  • Identify why metrics are important for IT
  • Identify the relationship among GISRA, NIST SP
    800-26, and IT Security Metrics
  • Describe IT Security Metrics
  • Describe metrics development process
  • Apply metrics development process by completing a
    Metrics Form for one of the OMB GISRA reporting
    requirements for FY02
  • Identify metrics-related Roles and
  • Describe how to implement a Metrics Program

Metrics Development
In this section, you will
  • Learn the definition and characteristics of IT
    Security Metrics
  • Identify the difference between Performance Goals
    and IT Security Metrics
  • Learn the seven-step IT Security Metrics
    Development Process
  • Discover the types of information and insights
    that can be gained from IT Security Metrics
  • Complete three examples of IT Security Metrics

What are IT Security Metrics?
  • IT Security Metrics are tools that facilitate
    decision making and accountability through
    collection, analysis, and reporting of relevant
    performance data.
  • Based on IT security performance goals and
  • Quantifiable
  • Obtainable/feasible to measure
  • Repeatable
  • Provide relevant performance trends over time
  • Useful in tracking performance and directing

Why Measure IT Security?
  • Government Information Security Reform Act
  • Clinger-Cohen Act
  • Government Paperwork Reduction Act (GPRA)
  • Satisfy regulatory requirements

  • Measure successes and failures ofpast and
    current security investments
  • Justify future investments
  • Enable investment targeting to identified
    areas in need
  • Ensure best value from security

  • Build confidence in leadership
  • Demonstrate improvement to stakeholders
  • Play key role in initiating improvementactions
    based on performance trends
  • Enable relevant, realistic, appropriatesecurity
    procedure modification
  • Improve accountability to stakeholders
  • Ensure appropriate level of mission support
  • Determine IT security program effectiveness
  • Improve customer confidence

IT Security Metrics should support IT security
goals and objectives
  • IT Security Performance Goals identify desired
    results of system security program
  • IT Security Performance Objectives enable
    accomplishment of goals by
  • Identifying strategic practices, defined by
    security policies, procedures, and controls
  • Directing consistent implementation of policies
    and procedures across the organization
  • IT Security Metrics monitor accomplishment of
    goals and objectives by
  • Quantifying the level of implementation of
    security control objectives and techniques for a
    system and the effectiveness and efficiency of
    the controls within the organization
  • Using analysis of collected IT Security Metrics
    to determine adequacy of security activities and
    make appropriate business decisions

Exercise Performance Goal or IT Security Metric?
IT Security Metric
Performance Goal
Program Officials understand the risk to systems
undertheir control and determine the acceptable
level of risk.
Percentage of system security plans that
areupdated annually.
Duties are separated to ensure least
privilegeand individuals accountability.
Percentage of systems with automated virus
Data integrity and validation controls are used
to provide assurance that the information has
not beenaltered and the system functions as
Metrics development is a seven step process
The focus of the metrics program depends on IT
security program maturity
Stakeholders and Interests
  • Anyone within an organization is an IT security
    stakeholder, though some functions have a greater
    stake than others
  • CIO
  • Program Manager/System Owner
  • Security Program Manager
  • Resource Manager
  • Training/Human Resources Personnel
  • Each stakeholder needs a set of metrics that
    provides a view of the organizations IT security
    performance within their needs, for a total of no
    more than 10-20 metrics per stakeholder
  • Many IT Security Metrics can be created to
    measure each aspect of the organizations IT
    security. Selecting the most critical elements
    of the organizations IT security program during
    metrics prioritization will make the program
    manageable and successful

IT Security Performance Goals and Objectives
  • IT security performance goals and objectives are
    expressed in the form of high level policies and
    requirements in many laws, regulations, policies,
    and guidance that describe the dimensions of an
    effective IT security program
  • Clinger Cohen Act
  • Presidential Decision Directives 63
  • Government Information Security Reform Act
  • OMB Circular A-130, Appendix III
  • Critical Elements within NIST Special
    Publication 800-26
  • Federal Information Security Compliance Audit
    Manual (FISCAM)

IT Security Policies, Guidance, and Procedures
  • Some Federal guidance and agency-specific
    policies and procedures provide more detailed
    information specific to the agency
  • NIST SP 800-12, 800-14
  • Agency-specific policy and guidance
  • Subordinate questions within NIST Special
    Publication 800-26

System Security Program Implementation
  • System Security implementation includes
  • Processes and procedures in place
  • Existing capabilities
  • Areas for improvement
  • Existing metrics
  • Existing data sources that can be used to derive
    metrics data
  • These may be documented in the following sources
  • System Security Plans
  • OMB Plan of Actions and Milestones (POAM)
  • Latest GAO and IG findings
  • Tracking of security-related activities
  • Risk assessments and penetration testing results

Metrics can describe three aspects of IT security
program operations and management
Business Impactcan be measured through
correlation analysis oncean organizations
processes are self-regenerating andmeasurement
data gathering is transparent.
Level of Implementation Most organizations are
new to measuring IT security with performance
metrics. They will begin by measuring level of
implementation of its security policies and
procedures. Instituting a metrics program is
the first step to process maturity.
Security Program Effectiveness and EfficiencyAs
an organizations process maturity increases and
performance data becomes more readily available,
metrics will focus on program efficiency and
It is important to record the specifics of each
metric for the purposes of data analysis and
possible metric reuse
Describes the overall functionality obtained by
collecting the metric
Defines the metric by describing the quantitative
measurement(s) provided by the metric
Specific questions that will need to be answered
via survey or through automatic data gathering to
be able to calculate the metric
Survey Question(s)
Proposes the periods for collection of data to be
used for measuring changes over time. Suggested
time periods are based on likely updates
occurring in the applicable process
Describes the calculation to be performed that
results in a numeric expression of a metric
Data Source
Lists the location of the data to be used in
calculating the metric
Provides information about the meaning of the
metric and its performance trend. Proposes
possible causes of trends, identified through
measurement, and points at possible solutions to
correct observed shortcomings
Metrics can help identify causes of poor
performance, including
  • Insufficient human, monetary, or other resources
    can be causing negative performance trends
  • Lack of appropriate training for the personnel
    installing, administering, maintaining, or using
    the systems
  • Security patches that have been removed during
    the operating system upgrades
  • New or upgraded systems that are not configured
    with all required security settings and patches
  • Security patches or upgrades that are
    incompatible with software applications supported
    by the system
  • Lack of management awareness and/or commitment
    to security
  • Lack of policies and procedures that are required
    to ensure existence, use, and audit of required
    security functions
  • Poor system and security architectures that make
    systems vulnerable
  • Inefficient planning processes that influence the
    metrics (including communication processes
    necessary to direct organizational actions)

System Upgrades
Configuration Management Practices
Software Compatibility
Awareness and Commitment
Policies and Procedures
Inefficient Processes
How does NIST SP 800-26 relate to metrics?
Critical Element
13.1. Have employees received adequatetraining
to fulfill their security responsibilities?
Subordinate Questions
13.1.3 Is there a mandatory annual refresher
13.1.1 Have employees received a copy ofthe
Rules of Behavior?
13.1.4 Are methods employed to make employees
aware ofsecurity, i.e., posters, booklets?
13.1.5 Have employeesreceived a copy ofor have
easy access toagency security procedures and
Implementation Evidence
of employees who signed employee agreements
of employees who received annual refresher
of new employees who underwent security
awareness training
. . .
Example 1 Security Awareness, Training, and
Education (Implementation)
Critical Element 13.1 Have employees received
adequate training to fulfill their security
responsibilities? Subordinate Questions 13.1.1
Have employees received a copy of Rules of
Behavior? 13.1.5 Have employees received a copy
of or have easy access to agency security
procedures and policies?
Percentage employees who underwent initial IT
security awareness training
To determine the number of new employees who
underwent required IT security awareness
training, including receiving a copy of Rules of
Behavior and security policies and procedures.
1. Is security awareness training required for
new employees?
2. Do employees receive a copy of Rules of
Behavior as a part of their training?
3. Do employees receive a copy of policies and
procedures or a summary with references as a part
of their training?
4. How many new employees joined your
organization during current past reporting
period? Fill in the blank____________ How many
of those took the training? Fill in the
If response to 1-3 was Yes ( who took
training) / (Total of new employees)
Data Source
Security awareness training tracking by the
responsible office, or manual or automated system.
Security awareness training is effective when it
includes specific information on rules of
behavior and information on security policies and
procedures. This metric validates the content of
training and determines the percentage of
employees who took the training. High numbers
close to 100 are highly desirable. Having
employees who do not understand the security
implications of their actions significantly
increases the number of security incidents caused
by insiders. This may also increase the
opportunities for unauthorized access to systems
caused by staff not following security policies
and procedures.
Example 2 Incident Response Capability
Critical Element 14.1 Is there a capability to
provide help to users when a security incident
occurs in the system? Subordinate Question
14.1.3 Are incidents monitored and tracked until
Elapsed time between when an incident is reported
and when affected partiesare notified of a
To determine how long it takes to notify users of
What is an average number of hours that it takes
to notify users that may be affected by a
reported incident?
Under 1
Under 24
Under 12
Over 24
At the program, level average time weighted
average among all responses
Incident Response Database
Data Source
Users should be notified of a problem as quickly
as possible. A delay in notification time may
point to a lack of incident monitoring and
tracking until resolution. Higher number points
to inefficiency in (or lack of) a notification
process. This metric can also be calculated as
percentage of all incidents that are communicated
to affected parties within 1 hour, 12 hours, 24
hours, and over 24 hours.
Example 3 Hardware and System Software
Maintenance (Implementation)
Critical Element 10.3. Are systems managed to
reduce vulnerabilities? Subordinate Question
10.3.2 Are systems periodically reviewed for
known vulnerabilities and software patches
promptly installed?
Percentage of systems with latest patches
To quantify the level of risk exposure caused by
the lack of current security patch implementation
  1. Is regular vulnerability scanning conducted?

2. How many components are scanned every time?
Fill in the blank ________ 3. How many of those
had up-to-date patches during the last scanning
cycle? Fill in the blank ________
4. If your answer to question 1 was no, what
was the reason?
Insufficient staff
Insufficient funding
Superseding other priorities
( components with up-to-date patches)/(Total
of components)
Data Source
Regular Vulnerability Scanning
This metric monitors compliance with applicable
patches and provides useful information about the
level of risk exposure at a system level. The
goal in this case is 100. The desired trend for
this metric is downward. Question 3 identifies
why a patch compliance validation process may be
lacking and points at specific corrective actions
to facilitate establishment of such process.
Metrics Development Criteria What is a Good
  • ? Based on IT security performance goals and
    objectives NIST SP 800-26 Critical Elements and
    Subordinate Questions are used to derive
    performance goals and objectives
  • ? Quantifiable Metrics should yield
    quantitative rather than qualitative information
    to increase the objectivity and validity of data
  • ? Obtainable/Feasible to measure Metrics data
    should be available or easily collected through
    interviewing or by accessing data repositories.
    If a metric requires significant modification of
    agency processes or implementing a new tool, data
    collection may not be feasible at this time
  • ? Repeatable Measurements should be able to be
    repeated in a standard way at predetermined
    intervals to identify trends or identify if
    positive changes have occurred as a result of
    corrective actions
  • ? Provide relevant performance trends over
    time Repeated measurements reveal change in a
    timely manner
  • ? Useful in tracking performance and directing
    resources Metrics should be useful to
    stakeholders and should yield information that is
    important in financial decision making

Breakout Session
Breakout Session
  • Goal To complete a Metric Form for one of the
    metrics that is required for GISRA reporting for
    FY 2002. This includes identifying the NIST SP
    800-26 Critical Element and Subordinate Question
    that map to the specific GISRA question from OMB
  • Duration 30 minutes
  • Method
  • Read the metric your Breakout Group is assigned
  • Select the NIST SP 800-26 Critical Element that
    includes your metric
  • Select the Subordinate Question within the
    Critical Element that maps to your metric.
    Remember, a single metric can use more than one
    Subordinate Question
  • Complete the Metric Forms sections, giving
    particular attention to what implementation
    evidence may exist that corresponds to your
    Subordinate Question
  • Follow up Each Group will have five minutes to
    brief their Form to the other groups. This
    brief should include
  • The metric your Breakout Group was assigned
  • The Critical Element and Subordinate Question
    that maps to your metric.
  • The completed Metric Form, including
    implementation evidence and indicators. A list
    of possible sources of the data you need to
    uncover for your metric

Critical Element Subordinate Question
Data Source
Metrics Program Implementation
In this section, you will
  • Receive an introduction to the IT Security
    Metrics-related roles and responsibilities
  • Learn the steps involved in IT Security Metrics
    program implementation by learning the process
    and following an example through the process

Multiple success factors can influence quality
and sophistication of IT Security Metrics (slide
1 of 2)
  • Ensure that IT Security Metrics Program is
  • Use no more than 10-20 metrics at a time, based
    on current priorities
  • Phase old metrics out and phase new metrics in
    when performance targets are reached or when
    requirements change
  • Ensure acceptable quality of data
  • Data collection methods and data repositories
    should be standardized
  • Events must be reported in a standard manner
    throughout the organization and the results of
    such reports need to be stored in the data

Multiple success factors can influence quality
and sophistication of IT Security Metrics (slide
2 of 2)
  • Obtain organizational acceptance
  • Metrics need to be validated with organizations
    stakeholders within headquarters and in the field
  • Metrics should be vetted through appropriate
    approval channels
  • Ensure that metrics are useful and relevant
  • Useful data should be collected
  • Not all data are useful

Metrics-related roles and responsibilities are
dispersed throughout an organization
Each organization will implement a metrics
program specific to its needs
  • Tailor to organization and business processes
  • Identify IT Security Metrics-related stakeholder
    roles and responsibilities
  • Lay out required infrastructure changes, such as
    creation of web-based data collection tools and
    of new data repositories
  • Identify required modifications of the current
    data sources
  • Define data reporting formats

Output from standard security activities can be
used to quantify IT security performance
  • Incident Handling
  • Testing
  • Network Management
  • Audit Logs
  • Network and System Billing
  • Configuration Management
  • Contingency Planning
  • Training
  • Certification and Accreditation

IT Security Metrics data collection must be as
transparent and non-intrusive as possible.
IT Security Metrics Program Implementation Process
  • Analyze collected data
  • Conduct gap analysis - Identify gaps between
    actual and desired performance
  • Identify reasons for undesired results
  • Identify areas requiringimprovement
  • Determine range of corrective actions
  • Select most appropriate corrective actions
  • Prioritize corrective actions based on overall
    risk mitigation goals
  • Develop cost model - Project cost for
    each corrective action
  • Perform sensitivity analysis
  • Develop business case
  • Prepare budget submission
  • Identify stakeholders
  • Determine goals / objectives
  • Review existing metrics
  • Develop new metrics
  • Identify data collection methods and tools
  • Collect metrics
  • Budget allocated
  • Available resources prioritized
  • Resources assigned
  • Management
  • Operational
  • Technical
  • Track progress
  • Report as required

Process Implementation Example
Lack of IT security refreshertraining may be
causing weakpasswords, identified bya password
cracker that isrun regularly.
Employees should be required to take annual IT
security refresher training as part of their
annual review process.
Since annual refresher training has ceased, the
number of weak passwords has increased by 50.
Only 5 of employees receive annual IT security
refresher training.
  • A budget submission detailingmetrics findings
    related to annual IT security refresher training
    was submitted, and funding received.

Annual refresher training,an operational
control,is instituted.
Since the training was re-instituted, the
percentageof weak passwords hasdecreased by 40
  • Discussed why Metrics are important for IT
  • Obtained understanding of the relationship
    between GISRA, NIST SP 800-26, and IT Security
  • Described IT Security Metrics
  • Described the Metrics Development Process
  • Created metrics to be implemented at a system
    level through applying metrics development
  • Discussed metrics-related Roles and
  • Described how to implement a Metrics Program

Next Steps
  • You can immediately use what you have learned
    today to propose some metrics within your agency
  • Notes of the workshop will be published in two
  • You can use the three metrics presented during
    the workshop and those that we developed together
    for your GISRA submission
  • Metrics Guidance first draft will be published by
    September 30, 2002
  • Please contact Marianne Swanson if you have any
    questions at,