WIA PERFORMANCE MEASURES AND STANDARDS: The WIASRD, Common Measures and Standards Negotiation Challenges

1
WIA PERFORMANCE MEASURES AND STANDARDSThe
WIASRD, Common Measures andStandards Negotiation
Challenges

• Christopher T. King
• Ray Marshall Center for the Study of Human
  Resources
• University of Texas, Austin
• ctking_at_uts.cc.utexas.edu
• 512/471-2186
• David W. Stevens
• The Jacob France Institute
• University of Baltimore
• dstevens_at_ubmail.ubalt.edu
• 410/837-4729
• April 22, 2003

2
BRIEFING TOPICS

• 1. Highlights from PY 2000 program outcome
  information in the
• WIASRD files from the seven ADARE Project
  states,
• focusing on the quality of the data elements.
• 2. Negotiated, actual and actual minus negotiated
  difference in
• PY 2000 performance data for the seven ADARE
  Project states.
• 3. Observations about the proposed common
  measures.
• 4. WIA performance standards negotiation
  challenges and
• opportunities (including pros and cons of
  regression modeling).
• 5. Other challenges that will follow
  reauthorization.

3
EMPLOYED IN QUARTER AFTER EXIT QUARTER

• The data element code choices are yes, no and
  not yet available
• Georgia, Illinois and Missouri did not use the
  not yet available code.
• The four ADARE Project states that used the not
  yet available code
• used it the following percent of the time
• Florida 44 percent
• Maryland 73 percent
• Texas 23 percent
• Washington 50 percent

4
USE AND SOURCE OF SUPPLEMENTAL DATA

• The data element code choices are used case
  management files and
• record sharing/matching
• Florida, Missouri and Washington did not report
  any use of
• supplemental data sources.
• Georgia reported only three instances of
  supplemental data use.
• Texas reported using supplemental data one
  percent of the time.
• Illinois and Maryland reported using supplemental
  data three
• percent of the time.

5
OCCUPATIONAL CODEof any job held since exit

• This information is to be reported if the
  individual is reported as employed
• in the quarter after exit.
• The information can be based on information
  derived from case management
• files, follow-up services or other sources.
• It is not necessary to wait until information on
  employed in quarter after exit
• is available.
• Florida, Georgia and Maryland used only the
  nine-digit DOT code.
• Illinois and Texas used only the five-digit OES
  code.
• Washington used both the DOT and OES coding
  taxonomies.
• Missouri used the five-digit or six-digit ONet98
  code.

6
ENTERED TRAINING RELATED EMPLOYMENT

• Two-thirds of the yes or no entries for this data
  element were recorded as
• a yes.
• The range of affirmative entries was from a low
  of 29 percent for
• Maryland to a high of 94 percent for Florida.
• The reported method used by Florida, Maryland,
  Texas and Washington
• to determine training related employment was
  other appropriate method.
• The reported method used most often by Georgia,
  Illinois and Missouri
• was a comparison of the occupational codes of
  the training activity
• and the job, but each of these three states
  also used a comparison of
• the industry of employment with the occupation
  of training using
• an appropriate crosswalk.

7
ENTERED NONTRADITIONAL EMPLOYMENT

• The nontraditional employment designation can be
  based on either
• local or national data.
• Six percent of the yes or no entries for this
  data element were reported
• as a yes.
• The range of affirmative entries among the seven
  ADARE Project states
• was from a low of one percent to a high of
  fifteen percent.
• Texas did not report yes or no entries for this
  data element.

8
TYPE OF RECOGNIZED EDUCATIONAL/OCCUPATIONALCERTIF
ICATE, CREDENTIAL, DIPLOMA OR DEGREEATTAINED

• Seven codes are provided. States and localities
  have flexibility in
• choosing the methods used to collect data
  documenting this data
• element.
• Each of the seven ADARE Project states reported
  award of some
• credentials in each of the six type of
  credential categories.

9
PY 2000 CORE MEASURES OF PERFORMANCESEVEN ADARE
PROJECT STATES

• The four Adult and Dislocated Worker performance
  measures are covered.
• Entered employment rate.
• Employment and credential rate.
• Retention rate.
• Earnings change
• Each of the four charts that follow flies in PY
  2000 negotiated, actual
• and actual minus negotiated performance measure
  values for the
• seven ADARE Project states.

10
QUESTIONS TO ASK WHEN LOOKING AT THECHARTS THAT
FOLLOW

• Do I know enough about the criteria for
  specifying each negotiated
• performance measure value to interpret the
  observed differences
• in these negotiated values among the seven ADARE
  Project states?
• Do I know enough about the data sources that were
  used to calculate
• the actual performance measure values to
  interpret the actual minus
• negotiated differences in these values among the
  seven ADARE Project
• states?
• What management and/or policy conclusions can I
  reach based on
• my answers to the previous two questions?
• Can I be confident in making incentive awards and
  imposing sanctions
• based on actual minus negotiated value
  differences?

11
Program Year 2000 (July 2000-June 2001) Entered
Employment Rate
12
Program Year 2000 (July 2000-June 2001)
Employment And Credential Rate
13
Program Year 2000 (July 2000-June 2001)
Retention Rate
14
Program Year 2000 (July 2000-June 2001) Earnings
Change
15
REVISITING THE QUESTIONS ASKEDHAVING LOOKED AT
THECHARTS

• Do I know enough about the criteria for
  specifying each negotiated
• performance measure value to interpret the
  observed differences
• in these negotiated values among the seven ADARE
  Project states?
• Do I know enough about the data sources that were
  used to calculate
• the actual performance measure values to
  interpret the actual minus
• negotiated differences in these values among the
  seven ADARE Project
• states?
• What management and/or policy conclusions can I
  reach based on
• my answers to the previous two questions?
• Can I be confident in making incentive awards and
  imposing sanctions
• based on actual minus negotiated value
  differences?

16
COMMON MEASURE ISSUESPerformance Measure Quality

• ENTERED EMPLOYMENT RATE
• Registration date
• Employed or not employed at registration
• Exit date
• Entered employment by the end of the first
  quarter after exit
• ISSUES
• Staff decision whether and when to register a
  customer
• Quality of employed or not employed at
  registration data element
• Unintended consequences of this measure
• Staff decision when to enter or allow automatic
  entry of exit date
• Use of supplemental data sources

17
COMMON MEASURE ISSUESPerformance Measure Quality

• EMPLOYMENT RETENTION RATE
• Employed first quarter after exit (regardless of
  employment status
• at time of registration)
• Employed second quarter after exit
• Employed third quarter after exit
• ISSUES
• Stakeholder interest in this measure
• Drill-down questions that will be asked
• Use of supplemental data sources
• Timeliness of availability for intended uses

18
COMMON MEASURE ISSUESPerformance Measure Quality

• EARNINGS INCREASE
• Earnings in second quarter prior to registration
• Employed in first quarter after exit
• Earnings in first quarter after exit
• Earnings in third quarter after exit
• ISSUES
• Stakeholder interest in this measure
• Drill-down questions that will be asked
• Number of pays in each reference quarter
• Use of supplemental data sources
• Timeliness of availability for intended uses

19
COMMON MEASURE ISSUESPerformance Measure Quality

• EFFICIENCY
• The dollar amount specification to serve as the
  numerator
• The number of participants figure to serve as the
  denominator
• ISSUES
• Stakeholder interest in this measure
• Drill-down questions that will be asked
• Quality of data elements
• Unintended consequences

20
COMMON MEASURE ISSUESPerformance Measure Quality

• PLACEMENT IN EMPLOYMENT OR EDUCATION
• Registration date
• Enrolled in secondary education at registration
• Exit date
• Not enrolled in post-secondary education at
  registration
• Not employed at registration
• Enrolled in secondary education at exit
• Employed in first quarter after exit
• In military service in first quarter after exit
• Enrolled in post-secondary education in first
  quarter after exit
• Enrolled in advanced training/occupational skills
  training in
• first quarter after exit
• CONTINUED

21
COMMON MEASURE ISSUESPerformance Measure Quality

• PLACEMENT IN EMPLOYMENT OR EDUCATION
• CONTINUED.
• ISSUES
• Stakeholder interest in this measure
• Drill-down questions that will be asked
• Quality/uniformity of data definitions and
  sources
• Cost of data collection
• Access to education records
• Timeliness of data availability for intended uses
• Unintended consequenceproliferation of
• credentials

22
COMMON MEASURE ISSUESPerformance Measure Quality

• ATTAINMENT OF A DEGREE OR CERTIFICATE
• Registration date
• Enrolled in education
• Exit date
• Attained a diploma, GED, or certificate by the
  end of the third
• quarter after exit
• ISSUES
• Stakeholder interest in this measure
• Drill-down questions that will be asked
• Access to education records
• Quality/uniformity of data definitions and
  sources
• Timeliness of data availability
• Unintended consequences

23
COMMON MEASURE ISSUESPerformance Measure Quality

• LITERACY OR NUMERACY GAINS
• ?

24
COMMON MEASURE ISSUESPerformance Measure Quality

• FIVE ISSUES ARE OF PARTICULAR IMPORTANCE AND
  CONCERN
• The accuracy and probable unintended consequences
  associated
• with the employed or not employed at
  registration data element
• The integrity and value-added of supplemental
  data use
• Selection of denominator and numerator
  definitions for the
• proposed efficiency measure
• The complexity and value-added of the placement
  in employment
• or education measure
• Expected unintended consequences associated with
  the
• attainment of a degree of certificate measure

25
PERFORMANCE STANDARD ISSUES

• THREE TOPICS ARE OF PARTICULAR IMPORTANCE
• State and Local Workforce Area Benchmarking
• The Census Bureau LEHD Program as a potential
  source of local
• demographic and economic activity information
  for discretionary
• use in negotiation of state and local
  performance standards
• Benchmarking of own performance over time
• Benchmarking against other similar states or
  Local Workforce Areas
• Return to regression modeling? Pros and cons
• CONTINUED.

26
PERFORMANCE STANDARD ISSUES

• Challenges Associated with Pursuing Continuous
  Improvement
• Integrity of state and local management
  information systems
• over time
• Continuity of data source availability and
  content over time
• Expected unintended consequences

27
PERFORMANCE STANDARD ISSUES

• Vulnerability to Unintended State and Local
  Actions
• Discretionary opportunities to define selection
  in criteria,
• assignment to service components criteria
  (including whether
• and when to use partner services) and timing of
  exit criteria
• Investment in staff development can reduce the
  frequency
• of some of the unwanted behaviors that will
  otherwise
• follow introduction of the common measures

28
OTHER CHALLENGES

• Occupations in demand
• Required registration of some customers
• Stakeholder interest in number of customers
  served