Changing Perspectives on Workforce System Performance

1
Changing Perspectives on Workforce System
Performance

• Data Validation
• Workforce Innovations
• San Antonio
• July, 2004

2
Purpose of Todays Session

• Update states and grantees on what is new in data
  validation
• Provide information on the findings to date from
  the first round of validation
• Allow states and grantees to provide feedback to
  ETA on how to improve the DV process

3
Role of Data Validation in ETA Performance
Assessment

• Data validation is a key component in overall
  performance strategy
• Program funding is being directly tied to
  reliable performance outcomes (performance budget
  integration)
• Data validation required by OIG and now being
  reviewed by GAO
• Data validation is integrated into reporting
• Validation tools are evolving to meet state needs

4
Programs Included in the Data Validation Effort

• Unemployment Insurance Benefits and Tax (UI)
• Workforce Investment Act (WIA)
• Trade Adjustment Assistance (TAA and NAFTA-TAA)
• Labor Exchange
• National Farmworker Jobs Program (NFJP)
• Indian and Native American Programs (INA)
• Senior Community Service Employment (SCSEP)
• Office of Apprenticeship, Training, Employment,
  and Labor Services (OATELS)

5
How Does Validation Work?

• Two separate processes are required to ensure
  that performance data is reliable
• Report Validation
• Data Element Validation
• ETA provides software to states and grantees that
  analyzes participant records

6
Report Validation

• Ensures that performance calculations are
  accurate
• DV software creates an audit trail for the
  numerator and denominator for each performance
  measure
• Classifying participant records into performance
  outcome groups enables non-technical staff to
  validate and analyze program outcomes

7
Data Element Validation

• Report will not be accurate if the data being
  used by the software are wrong
• Requires checking data elements against source
  documentation to verify compliance with federal
  definitions
• Handbooks contain instructions and examples of
  acceptable source documents for each data element
  validated
• States identify state-specific source
  documentation to reflect the variability of state
  MIS systems and state/local documentation
  standards

8
Reporting of Validation Results

• Data validation software produces
• Report validation summary
• Data element validation summary and analytical
  reports
• WIA and LX software creates files with the annual
  report validation values for upload to ETA

9
Validation Efforts to Date

• Many states have shared their validation results
• First round of validation was a valuable learning
  experience for all
• ETA has not set standards for acceptable data
  quality
• Standards will be set for PY 2004 data validation

10
What is New for PY 2003 Validation

• New schedule for reporting and validation
• New software
• New policies for collection and retention of
  source documentation

11
Schedule for Reporting of Validation Results

• WIA RV will be due October 1, 2004 when the
  annual report is due
• WIA and TAA data element validation will be due
  February 1, 2005
• LX report validation will be due November 15,
  2004 with the report.

12
Data Validation for National Programs

• NFJP validation to begin in February
• Reports due in June
• Pilot of process was conducted in spring
• Software will be tested further in fall
• Training session scheduled for November
• Data validation to be added to SCSEP in late 2005
• Indian and Native American validation will be
  incorporated into existing reporting software

13
Revised Software

• WIA Version 3.0 to be released in mid-August
• New versions TAA (version 1.3) and LX (version
  1.8) validation software
• All will include automated upload of DV reports
  to ETA

14
WIA Software Changes

• Calculate performance for the new reporting
  periods
• Calculate Table O
• More complete edit checks
• Ability to filter source table and performance
  outcome groups to provide greater analytical
  flexibility
• Accept records for participants served only by
  NEGs

15
Software Upgrades for WIA Data Element Validation

• Revised Data Element Validation Worksheets to
  reflect reduction in elements
• Improved ability to identify records that have
  not been validated
• Ability to identify sampled records that have are
  missing, invalid, wrong SSN, or whose location is
  unknown.
• Ability to trace exported samples

16
States Experiences with Data Validation

• States had to determine staff to be responsible
  for data validation
• Communication of expectations and requirements to
  local areas
• Mode of data element validation onsite,
  centralized or both

17
State and Local Roles and Responsibilities

• States had varying experiences in identifying
  validation assignments
• Some states had no problem
• Some states took time to sort through roles of
  different units
• Some states still have not clarified assignments
  (particularly for TAA)
• Organization of case files at local areas was
  often not standardized or adequate

18
Improving the Clarity of Source Documentation
Requirements

• ETA has not had clear and specific policies for
  collection and retention of source documentation
• States need to provide clear guidance to local
  areas
• ETA will clarify requirements in change 1 to TEGL
  3-03
• Currently in clearance
• To be issued in August

19
Streamlined Data Element Validation Requirements

• As a result of state feedback, ETA reviewed and
  reduced the number of elements to be validated
• All elements directly related to performance or
  eligibility

20
Detail for reduction in elements
Program of Elements for PY02 of Elements for PY03
Adult 41 26
Dislocated Worker 44 30
Older Youth 48 35
Younger Youth 100 29
Trade 35 23
NFJP 17 14
Total 285 157
21
Various Methods for Data Element Validation

• Onsite validation is essential to preserve the
  integrity of the process
• Ideal for state staff to perform validation
  onsite
• Promotes communication and mutual understanding
• In some cases, onsite validation is impractical
• Distances are too great
• Small number of records
• States can therefore pursue a combination of
  onsite and remote validation if necessary

22
Findings for WIA Report Validation

• States had problems in two areas
• Problems with extract file imported into software
• Problems with calculations

23
WIA Report Validation -- File Problems

• Extract file imported into the software is
  incorrect
• Data in extract file does not match data in
  states data system
• Inconsistent data
• File is different from the file used to calculate
  the report submitted to ETA
• Missing records
• Changed/Updated Data

24
WIA Report Validation Calculation Problems

• States excluded older Youth in advanced
  training/post-secondary school from performance,
  even if the youth is employed.
• Failure to distinguish pre-dislocation earnings
  from pre-registration earnings for dislocated
  workers
• Exclusion of records for earnings calculations
  due to 99,999.99.

25
Data Element Validation Findings

• Significant number of errors error rates
  exceeded 20 for some elements
• Many errors can be explained by lack of clarity
  in expectations for local source documentation
• Problems with changing wage records and WRIS data

26
WIA Data Element Validation Results for Adults
gt20 10-20 5-10 lt5
Individual with Disability Date of First Training Service Date of Birth Employed 1st qtr after Exit
Veteran Status Date of First Intensive Service Employment Status at registration Wages second quarter prior to registration
Other Reasons for Exit Source of Supplemental Data for 1st and 3rd Quarter Low Income Wages first qtr following exit quarter
Wages third quarter prior to registration TANF Wage third quarter following exit
Type of recognized certificate/credential Date of WIA Exit
27
WIA Data Element Validation Results for
Dislocated Workers
gt20 10-20 5-10 lt5
Displaced Homemakers Individual with a disability Date of Birth Employed 1st qtr after Exit
Veteran Status Date of actual qualifying dislocation Wages 2nd and 3rd quarter prior to registration
Date of First Training Service Date of WIA registration Wages 2nd and 3rd quarter prior to dislocation
Source of Supplemental Data for 1st and 3rd Quarter Date of WIA Exit Wages 1st, 2nd, and 3rd quarter following exit quarter
Type of recognized certificate/credential Employed in the 3rd quarter after exit
28
WIA Data Element Validation Results for Older
Youth
gt20 10-20 5-10 lt5
In post-secondary education or advanced training 3rd quarter after exit Veteran Status Employment status at registration Date of birth
Date of First Training Service Date of WIA Exit Wages 2nd and 3rd quarter prior to registration
Type of recognized certificate/credential Employed in the 3rd quarter after exit Wages 1st, 2nd, and 3rd quarter following exit quarter
In post-secondary education/advanced training 1st quarter after exit Source of supplemental 3rd quarter after exit Source of supplemental 1st quarter after exit
Other reasons for exit
29
WIA Data Element Validation Results for Younger
Youth
gt20 10-20 5-10 lt5
Date of WIA Exit Education Status at Registration Date of birth
Other reasons for exit
Date goals attained
Attained secondary sechool diploma
Date attained diploma
Youth Placement and Retention Information
30
TAA Data Element Validation Results by the
Numbers
gt20 10-20 5-10 lt5
Most recent qualifying separation Date of registration Employed in the 1st and 3rd quarters after exit
Date of Exit Earnings 1st, 2nd, and 3rd quarters after exit
Earnings 2nd and 3rd quarters prior to most recent qualifying separation
31
Continuing Challenges

• State wage record files are always changing
• One solution is to freeze the file to avoid
  changes
• States should track changes in order to validate
  wages
• Confidentiality of WRIS data
• WRIS has rules restricting access to information
• Software allows states to suppress display of
  wage values

32
Future of Data Validation

• Standards for acceptable error rates to be
  established
• ETA is moving toward a consolidated reporting
  system
• Data Validation will be integrated into the new
  reporting system

33
For More Information

• Contact Information
• Traci Di Martini
• 202-693-3698
• Dimartini.traci_at_dol.gov
• MPR Technical Assistance
• William Borden 609-275-2131
• Jonathan Ladinsky 609-275-2250
• WIATA_at_mathematica-mpr.com
• TAATA_at_mathematica-mpr.com
• ESTA_at_mathematica-mpr.com
• http//www.doleta.gov/Performance/reporting/tools_
  datavalidation.cfm

34
We Need Your Feedback

• Tell us about your experiences with Data
  Validation
• What did you learn that may help others
• What improvements can be made