Test Validation Litigation:

1
Test Validation Litigation

• Legal Requirements and Lessons Learned

2
Federal Law and Regulations

• Title VII of the Civil Rights Act of 1964, 42 USC
  2000e
• Uniform Guidelines on Employee Selection
  Procedures
• Cases interpreting the foregoing

3
Title VII

• Title VII of the Civil Rights Act of 1964 (as
  amended in 1991) prohibits basing employment
  decisions on race, gender, ethnicity, religion,
  or national origin
• Employment decisions include Recruitment,
  Hiring, Promotion, Transfer, Wages, Leave,
  Training, Discipline, and Termination

4
Title VII

• Has been interpreted to require that an
  employers selection procedures not result in
  disparate impact against a protected group,
  unless the procedure is demonstrated to be valid
  and consistent with business necessity

5
Title VII

• Selection procedure that results in adverse
  impact is presumed to be discriminatory, unless
  it is shown to be valid pursuant to the Uniform
  Guidelines.

6
Uniform Guidelines

• Issued jointly by the EEOC, DOJ, OFCCP and other
  federal agencies in 1978
• Set forth standards for determining adverse
  impact
• Set forth standards for demonstrating validity of
  selection procedures
• Include important record-keeping requirements

7
Litigation

• Plaintiff or class of plaintiffs allege that
  selection procedures have adverse impact and are
  not valid
• Plaintiffs must show, usually in a statistical
  fashion, that the proportion of the protected
  class hired or promoted was significantly less
  than would be expected
• Raises a rebuttable inference of discrimination

8
Litigation

• In general, plaintiffs can establish a prima
  facie case of adverse impact by relying upon
• The 80 rule or
• A statistically significant difference, usually
  expressed as standard deviations

9
Litigation

• Disputes over statistical evidence
• Applicant pools and applicant flow
• Aggregation of data across jobs, facilities
• Inclusion or exclusion of variables
• What statistical tests to apply based on sample
  sizes, etc.

10
Litigation

• Disputes over applicant data will be multiplied
  after adoption of the new QA
• Whether the employer acted to fill a position
• Whether the putative applicant complied with
  employers application process
• Whether the advertised requirements were valid
• Whether criteria mentioned in advertisement were
  covered requirements

11
Internet Applicants

• EEOC and OFCCP have proposed new Uniform
  Guidelines QA to define Internet Applicant
• OFCCP has just issued a new rule defining
  Internet Applicant and implementing new
  recordkeeping requirements
• May have broad impact on adverse impact disputes

12
Proposed QA

• In order for an individual to be an applicant in
  the context of the Internet and related
  electronic data processing technologies, the
  following must have occurred
• The employer has acted to fill a particular
  position
• The individual has followed the employer's
  standard procedures for submitting applications
  and
• The individual has indicated an interest in the
  particular position.

13
Proposed QA

• Q Are all the search criteria that employers use
  subject to disparate impact analysis?
• A Yes. All search criteria used are subject to
  disparate impact analysis. Disparate impact
  analysis can be based on Census or workforce
  data. If a disparate impact is shown, the
  employer must demonstrate that its criteria are
  job-related and consistent with business
  necessity for the job in question. 42 U.S.C.
  2000e-2(k).

14
EEOC Guidance

• Search parameters
• An employer who uses specific qualifications to
  search a database in order to narrow the field
  will be required to assess whether those
  qualifications have an adverse impact

15
OFCCP Rule

• The expression of interest indicates that the
  individual possesses the basic qualifications
• Advertised qualifications
• Qualifications established in advance

16
OFCCP Rule

• Basic qualifications must be
• Non-comparative
• Objective
• Job related

17
Litigation

• Once plaintiffs establish adverse impact, burden
  shifts to employer to demonstrate validity
• Expert testimony in support of existing validity
  evidence
• New retrospective validation study

18
Litigation

• Plaintiffs counter with expert testimony
  attacking the validity evidence
• Compliance with Uniform Guidelines
• Compliance with professional standards
• Alternatives with lesser adverse impact

19
Litigation

• Aspects of the selection process that are subject
  to challenge
• Recruiting and advertising
• Application process and applicant definition
• MQ or other preliminary screening
• Examination, as a whole and by item
• Scoring
• Use of scores
• Final decisionmaking process

20
Litigation

• Aspects of validation that are subject to
  challenge
• Sampling plan and selection of SMEs
• Qualification of SMEs
• Interaction with SMEs
• Analysis of SME input
• Qualifications of development staff
• continued

21
Litigation

• Design of test instrument and items
• Item ratings and item bias
• Consideration of alternatives
• Compensatory vs. non-compensatory
• Alternative use of scores
• Consideration of alternative procedures
• Documentation
• More documentation

22
Reynolds v. ALDOT

• Litigation concerning the validity of minimum
  qualifications for several job classes
• Pursuant to consent decree, defendants had the
  burden of demonstrating validity in response to
  objections made by plaintiffs

23
Reynolds v. ALDOT

• Disputes concerning overall MQ development
  process
• Disputes concerning particular job classes
• Engineering jobs (including mgmt)
• Right of Way jobs

24
MQ Development Process

• Gather and analyze background information
• Prior selection procedures for same job
• Data from other employers
• ONET

25
MQ Development Process

• Select subject matter experts
• Job incumbents and supervisors with a minimum
  level of experience in the job
• Representation of
• Race
• Gender
• Functional areas
• Geographic locations
• Supervisors and incumbents

26
MQ Development Process

• Draft MQ Development form
• Used by SMEs as a tool for organizing thoughts
  about tentative MQ statements

27
MQ Development Process

• SME Meeting
• Review list of qualifying KSAs from job analysis
• Define minimum qualification and discuss
  purpose
• Develop and discuss tentative MQ statements

28
MQ Development Process

• SMEs are instructed that
• MQs are designed to identify the barely
  acceptable applicant
• MQs are the minimal levels of education,
  training, prior work experience or other
  attribute that would be necessary to acquire the
  KSAs needed to perform at a minimally acceptable
  level on Day 1

29
MQ Development Process

• SME-developed tentative MQ statements are
  bracketed with statements requiring lesser and
  greater amounts of the same type of qualification

30
MQ Development Process

• SME rating session
• SMEs rate the proposed MQ statements for
  suitability and link them to the surviving KSAs

31
MQ Development Process

• Suitability rating
• 0 not at all
• 1 not enough to expect
• 2 appropriately defines
• 3 too much to expect

32
MQ Development Process

• Linkage ratings
• Dichotomous (yes/no) scale
• Can the particular K, S, or A be acquired from
  the experience described in the proposed MQ
  statement?

33
MQ Development Process

• SME supplemental information questionnaire
• Solicits suggestions for any alternative MQs
• Solicits SME input concerning substitution of
  experience for education, or vice versa

34
MQ Development Process

• SME ratings analyzed by I/O Psychologists
• Mean ratings closest to 2.0
• Number of 2.0 ratings
• Number of KSAs to which statement linked at .50
  or higher
• Mode of combined ratings
• Adverse impact data
• Supplemental SME info

35
MQ Development Process

• Final MQ statements selected using professional
  judgment after consideration of all these factors

36
Challenges

• Plaintiffs claimed
• Education and experience MQs are not capable of
  content validation
• The MQs were not sufficiently specific
• Compound MQ statements are not permissible
• SME ratings and rating scales were improper
• continued

37
Challenges

• Necessary at entry ratings didnt include those
  who said they didnt use the K, S, or A
• Use of the same SMEs to develop and rate
  tentative MQ statements was improper
• Documentation did not satisfy the requirements of
  the Guidelines
• continued

38
Challenges

• Defendants failed to properly consider
  alternatives with lesser adverse impact
• MQ development form was improper
• Low pass rates indicate MQs too difficult
• SMEs not properly instructed on how to identify
  barely acceptable applicant

39
Defense

• Lay the foundation
• Project manager testifies to establish procedures
• Individuals responsible for development work
  testify about what was done
• All necessary documentation introduced

40
Defense

• Documentation is crucial
• Demonstrates compliance with Guidelines and
  professional standards
• Corroborates lay witness testimony as to what was
  done and why
• Documentation should include professional
  literature supporting the methodology

41
Defense

• Expert Witnesses
• Experts file report before trial supporting
  validity
• Following lay witnesses, experts testify that
  procedures and work done comport with Guidelines
  and professional standards
• Experts stress the exercise of professional
  judgment

42
Findings

• The MQ development process is consistent with the
  requirements of the Guidelines and leads to
  content valid MQs

43
Findings

• The Uniform Guidelines operate, of necessity, at
  a general level. The specific mechanisms for
  complying with the requirements of the Guidelines
  are left to the professional judgment of those
  responsible for their implementation.

44
Findings

• In making determinations concerning compliance
  with the Uniform Guidelines, it is appropriate
  for the Court to rely on the testimony of experts

45
Findings

• Defendants documented their MQ development
  efforts in a manner consistent with the Guidelines

46
Findings

• It was appropriate to validate compound MQ
  statements, where the requirements are designed
  to work together as a single statement.

47
Findings

• It was appropriate to allow SMEs to rate more
  than one tentative MQ as appropriately defining
  the barely acceptable applicant, because the SMEs
  could judge that there was a range of acceptable
  MQs in the series

48
Findings

• Alternative selection procedures must be
  considered only where they have
• Lesser adverse impact, AND
• Substantially equal validity

49
Findings

• Plaintiffs must proffer specific alternatives
  that they claim should have been considered
• Plaintiffs have the burden of showing that any
  alternative proffered by them would be of
  substantially equal validity and have lesser
  adverse impact

50
Findings

• It was appropriate to exclude from the
  calculation of necessary at entry ratings those
  SMEs who said they did not currently use a K, S,
  or A

51
Findings

• Defendants procedures for selecting SMEs were
  appropriate and in compliance with the Guidelines
• Guidelines do not require the use of any
  particular sampling methods

52
Findings

• The MQ Development Form was an appropriate tool
  for helping SMEs to organize their thoughts and
  begin thinking about MQ statements

53
Findings

• SMEs were given a sufficient definition of the
  terms minimum qualification and barely
  acceptable applicant

54
Findings

• There was ample evidence to support the
  defendants use of a dichotomous rating scale

55
Findings

• There was ample evidence to support the
  defendants use of a .50 linkage screen

56
Findings

• The existence of a validated selection procedure
  obviates the need for a separate consideration of
  business necessity

57
How to

• Avoid Trouble

58
Documented Procedures

• Develop standardized procedures to be followed in
  development projects
• Document standardized procedures in a manual
• Leave room for exercise of professional judgment

59
Documented Procedures

• Why are written procedures desirable?
• Ensure that everyone has the same understanding
  of the process
• Corroborate testimony concerning the process
• Refreshes the memory of a witness concerning work
  done long before testimony
• Enhances courts confidence in the process

60
Documented Procedures

• In developing standard procedures, review
  literature
• retain documentation of support for your
  procedures
• If there is conflicting literature, consider and
  document reasons for departing from that
  literature

61
Documented Procedures

• In developing standard procedures, consider legal
  precedent
• Become familiar with case law
• Consult with counsel

62
Documented Procedures

• Dont go out on a limb
• Take the most well supported route if possible
• Dont use untested or experimental procedures
  unless theres no other reasonable choice

63
Documented Procedures

• If litigation is ongoing or anticipated, involve
  outside expert early
• Expert should contribute to, or review
  procedures
• Expert should review own prior writings and
  testimony to ensure there is no significant
  conflict

64
Documented Procedures

• Pay careful attention to the requirements of the
  Uniform Guidelines
• The Uniform Guidelines are federal regulations,
  and they remain an important legal standard that
  courts may apply to determine compliance with the
  law

65
Staffing Development Projects

• Limit the number of staff members
• A small number of highly trained and well
  qualified people should perform most development
  work
• To the extent that less experienced assistants
  are used, they should be very closely supervised
  and not make judgment calls

66
Staffing Development Projects

• Carefully consider the background of each staff
  member
• Prior development experience
• Education
• Training

67
Staffing Development Projects

• Carefully consider the ability of each staff
  member to act as a witness
• Temperament
• Communication skills
• Attention to detail
• Presence

68
SME Selection

• SMEs should be unbiased and knowledgeable
• On the job for a reasonable length of time
  (ideally, not less than six months)
• Balance incumbents with a reasonable number of
  supervisors
• Demographic representation (random sample not
  required)
• Exclude those with demonstrable bias

69
Selecting an Expert

• The basic considerations
• Education
• Practical, hands-on development experience
• Experience as an expert witness

70
Selecting an Expert

• Other important considerations
• Review prior testimony, reports, and court
  opinions
• Experience in developing or critiquing selection
  processes for similar jobs
• Experience in developing or critiquing selection
  procedures of the types at issue
• All prior writings, as they may relate to any
  anticipated issue in the case at hand
• continued

71
Selecting an Expert

• Experience with or knowledge of other persons
  involved
• Thorough discussion and a common understanding of
  the experts role in the process, including
  interaction with other parties or their experts

72
During Development

• Carefully document all work
• The smallest, most seemingly insignificant detail
  may be raised as a problem by the other party
• Meticulous record-keeping can carry the day
• Consider relying on provisions of Uniform
  Guidelines as a point by point outline for
  documentation

73
During Development

• Exception to the meticulous documentation rule
• Dont correspond with staff in writing about
  questions and problems, if a telephone
  conversation or meeting will suffice
• You can always document your resolution of a
  problem with a memo afterwards

74
During Development

• When a deviation from your normal practices is
  called for, document it
• Provide an explanation for the decision to change
  the process
• Include any support in the professional literature

75
During Development

• Consideration of Alternatives
• User must undertake a reasonable investigation of
  alternatives that may be of substantially equal
  validity and lesser adverse impact
• Professional literature
• Other procedures used for same or similar jobs,
  by the user or by other employers

76
During Development

• Consideration of alternatives issue arises in
  litigation in two ways
• Plaintiffs may complain that the user has failed
  to consider suitable alternatives as required
• Plaintiffs may proffer a specific alternative
  that they allege is of substantially equal
  validity and lesser adverse impact

77
During Development

• Review the recruitment efforts that will be made
• Ensure that announcements and advertisements
  dont set forth qualifications that have not
  been validated
• Review the scope of recruitment to ensure that it
  is not restricted in ways that may be found
  discriminatory

78
During Development

• Avoid claims concerning the use of scores, which
  may arise even where the procedures themselves
  are valid
• Weighting of multiple components
• Compensatory vs. non-compensatory use of
  component scores
• Rank ordering of scores

79
During Development

• Weighting of multiple components
• Plaintiffs may contend for a different weighting
  scheme that they allege reduces between-group
  differences
• Carefully document rationale for chosen procedure
• Support with generally accepted principles,
  articles, and professional texts, where possible

80
During Development

• Compensatory vs. non-compensatory use
• Issue usually arises when some components deal
  with hard skills, while others test soft
  interpersonal skills that may have lesser adverse
  impact
• Job analysis information must support
  non-compensatory use
• Be prepared to articulate how job analysis
  supports such use

81
During Development

• Rank order scoring (including, but not limited
  to, top down selection)
• User must be able to show that a higher score on
  a content valid procedure is likely to result in
  better job performance
• Ask SMEs to rate the extent to which each
  exercise distinguishes between levels of
  performance

82
During Development

• Final Validation Report
• The documentation process should culiminate in a
  comprehensive written report
• Describe work performed
• Describe resulting products
• Discuss decisions made and professional judgment
  exercised

83
During Development

• Beware of deadlines!
• If you must agree to a deadline, decide how much
  time you are comfortable with, and then add some
  more time for the unanticipated
• Let the court and/or other parties know of
  potential obstacles or delays as soon as you know
• Ensure that ways to extend the deadline are set
  out

84
During Development

• Establish a review procedure for all development
  work
• One or two people provide final review of all
  completed work
• Reviewers will be witnesses

85
Be Prepared!

• Dont wait until the wolves are at your door to
  find out if your house is made of straw or bricks
• Consider a legal compliance review to uncover
  vulnerabilities while you have the luxury of
  correcting them

86
Lisa W. Borden

• Baker, Donelson, Bearman,
• Caldwell Berkowitz, P.C.
• 1600 SouthTrust Tower
• Birmingham, Alabama 35203
• [phone]
• lborden_at_bakerdonelson.com