Performance Measurement Tools for Justice Information Technology Projects

1
Performance Measurement Tools for Justice
Information Technology Projects

• Illinois/ Cook County
• Focus Group Experience

2
Using scenario approach to reach agreement and
define performance

• Bring stakeholders together to develop a criminal
  justice scenario
• Reach consensus on the desired state of
  integration
• Define the current state of integration
  (baseline)
• Quantify gap between current state and desired
  state
• Define desired outcomes
• Develop objectives and performance measures

3
IIJIS Strategic Plan Existing performance
measures

• 2.1.4. Number of mechanisms identified to reduce
  paper-based processes
• 5.2.4. Number of stakeholders adopting functional
  standards promoting interoperability by September
  2003
• 5.2.5. Number of stakeholder agencies recognized
  through the certification program
• 6.1.3. Percent of recommended infrastructure
  solutions implemented
• 6.3.3. Year 1 Number of users utilizing the
  resource center
• 6.3.4. Year 2 Percent increase of users
  utilizing the resource center

4
IIJIS Strategic Plan Existing performance
measures

• 6.4.5. Year 2 Percent increase of stakeholders
  adopting enterprise-wide disaster recovery plans
• 6.4.6. Year 1 Number of stakeholders performing
  disaster recovery tests
• 6.4.7. Year 2 Percent increase of stakeholders
  performing disaster recovery tests.
• 7.3.1. Number of research projects on biometric
  technological solutions completed by September
  2004
• 7.3.2. Number of research projects on costs and
  benefits of biometrics completed by September
  2004

5
Illinois/Cook County Focus Group on Performance
Measures

• Sixth in a series of facilitated workgroups by
  CSLJ to refine validate PMs
• All participants had contributed to development
  of state and/or Cook County strategic plans
• Participants were provided information on PM
  Project and the PM toolkit
• Participants applied toolkit to IIJIS Strategic
  Plan then answered questions and shared their
  reactions with CSLJ staff

6
PARTICIPANT CONCLUSIONS Performance measures
cannot be objectively selected stakeholders must
reach consensus

• Stakeholders, depending on their role within the
  justice enterprise, are likely to have strongly
  divergent views
• Higher-level goals and outcomes will usually be
  easier to agree upon than specific performance
  measures
• Performance measure selection, however, is not
  entirely subjective, and some measures are likely
  to be agreed upon by all

7
PARTICIPANT CONCLUSIONS It is nearly impossible
to definitively establish causal linkages to
public safety outcomes

• Outcomes become more diffuse the further out you
  get from actual processes
• Improved information sharing is only one of a
  number of factors affecting public safety
• A single performance measure may be inadequate to
  indicate that a system is working (or not)
• A carefully selected family of measures is
  preferable, and should be thoroughly tested
  before being deployed.

8
PARTICIPANT CONCLUSIONS Public sector outcome
measures are driven by different factors than for
the private sector

• Greater integration success by private industry
  due to clearer lines of accountability
• While the private sector is driven by the bottom
  line, government is driven by the competing needs
  of many agencies with differing missions and
  purposes lines of accountability are therefore
  more diffuse.
• Government is accountable not only to the
  governor, legislature, and public officials, but
  to the public as well
• Concern that the public believes that the justice
  system works as it is portrayed on television
  creates another type of accountability

9
PARTICIPANT CONCLUSION Performance measures
derived from strategic planning are different
from those derived from tactical planning

• Strategic plans are often written at a a very
  high level in order to convey the conceptual
  framework (particularly in state-level plans)
• Strategic performance measures often address
  pre-implementation process activities, thus
  producing a preponderance of outputs rather
  than outcomes
• Two-step planning process could make logic
  model/theory of change more straightforward

10
Performance measurement caveats

• Most people (including your employees and
  consultants) can learn to make the measures come
  out the way they think you want them to, while
  exerting a minimum of effort in actually
  improving a process
• Always question the measures youve defined,
  keeping in mind that the people applying them
  could find ways of boosting the measures without
  really improving anything
• Test each measure to determine if it operates as
  expected. Does it always go one way when things
  get better and the other when things get worse?

11
Performance measurement musts

• Early and often, measure and evaluate progress
  toward the goals and objectives that have been
  defined by the governing body
• Continually verify that defined measures actually
  correspond with the achievement of goals and
  objectives
• Resist overreaching measures of long-term,
  global outcomes tend to be unreliable.
• Performance measurement is complex and requires
  significant expertise

12
Strategic Goal 3 Identify and recommend
cost-effective biometric identification
applications Objective 3.1 By September 2004,
research, identify, and recommend technological
applications that support biometrics for rapid
identification. Objective 3.2 By September 2004,
research, identify, and evaluate the costs and
benefits of biometric identification
applications. Outcomes Increased knowledge of
biometric technologies Improved cost-effective
biometric identification solutions Performance
Measures Number of research projects on
biometric technological solutions completed by
September 2004 Number of research projects on
costs and benefits of biometrics completed by
September 2004 Number of research reports
presented to the IIJIS Governing Body
13
Revised Performance measures for Strategic Goal
3 Assumed Causal Chain as Inputs, Outputs, and
Outcomes
Staff work time conducting research on biometric
solutions
Input
Increased knowledge of biometric technologies
Output
Increased knowledge of cost-effective biometric
solutions for courtroom identification of
defendants
Output
Eliminate problem of sentencing wrong individual
Short-term Outcome
More accurate reporting of court dispositions to
criminal history repository
Short-term Outcome
More complete criminal history records
Intermediate Outcome
More informed justice decision-making
Intermediate Outcome
Enhanced Public Safety
Final Outcome
14
Why do I have to worry about all this?

• Projects have been de-funded to lack of
  performance measure data
• Need to justify capital expenditures
• Need for public accountability
• Its part of professional tool kit