PSI Division of Work and Occupational Psychology - PowerPoint PPT Presentation

Loading...

PPT – PSI Division of Work and Occupational Psychology PowerPoint presentation | free to view - id: c39ef-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

PSI Division of Work and Occupational Psychology

Description:

Situational Judgement Exercises used since mid 1990's ... While you are on patrol, a youth who is one of a group throws an empty mineral can ... – PowerPoint PPT presentation

Number of Views:41
Avg rating:3.0/5.0
Slides: 31
Provided by: lyndash
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: PSI Division of Work and Occupational Psychology


1
  • PSI Division of Work and Occupational Psychology
  • CPD Seminar February 2008
  • Recruitment and Selection
  • Implementing New Approaches
  • Áine Gray
  • Public Appointments Service

2
Our Customers
  • Civil Service
  • Entry to the civil service
  • Promotions
  • Local Government
  • Senior Management
  • Professional
  • Technical
  • Health Sector
  • Senior Management
  • Professional
  • Technical
  • An Garda Síochána

80 100 Clients
Handling approx. 80,000 applications p.a.
Filling approx. 5,000 vacancies p.a. Emerging
RD capability
3
Overview
  • Realistic Job Simulations
  • Situational Judgment exercises
  • Video based tests
  • Internet Based Testing
  • Case Study Using the Internet to assess people
    with disabilities

4
Realistic Job Simulations
  • Growing body of research evidence that realistic
    job simulations are effective in measuring job
    relevant skills
  • Also known as situational tests
  • Anastasi and Urbina (1997) define a situational
    test as one that places the test taker in a
    situation closely resembling or simulating a
    real-life criterion situation
  • Generally fairer than traditional cognitive
    tests, except where the job has a heavy cognitive
    load
  • Gives candidates a realistic job preview
  • Credible with candidates and the media

5
Some jobs are harder to simulate than others……..
6
(No Transcript)
7
(No Transcript)
8
What is a Situational Judgment Exercise (SJE)?
  • Also known as Situational Judgement Test, Job
    Simulation or Situational Judgement Inventory
  • First used in 1920s
  • Present candidates with job related situations/
    dilemma
  • Adopt a multiple choice format
  • May be bespoke for a particular role or
    off-the-shelf for particular grade e.g.
    managerial role

9
Experience of Public Appointments Service
  • Situational Judgement Exercises used since mid
    1990s
  • Used for a range of roles including entry level
    posts (Garda Trainee, Executive Officer, Junior
    Diplomat) and middle to Senior Management
    Positions (Assistant Principal recently
    introduced at Principal Officer level)
  • Used as part of a competency based approach to
    recruitment to assess a range of competencies
  • Particularly valuable because of gender
    performance
  • Almost all SJEs are bespoke for the role / grade
    and set in a Civil Service Context
  • Designed in-house with job incumbents, line
    mangers and other SMEs, trialling and item level
    analysis of test performance

10
  • You are on duty alone in a housing complex which
    has a reputation as a trouble spot.
  • While you are on patrol, a youth who is one of a
    group throws an empty mineral can
  • which hits you on the head. You are not
    physically injured, but are annoyed. The group
  • jeers at you in a threatening way.
  • What do you do?
  • A Notify the Station by radio that a situation is
    developing and then approach the group yourself
    to apprehend the can-thrower.
  • B Approach the group and warn them that they will
    be arrested if they continue being abusive.
  • C Choose not to respond to this incident as you
    are not injured by the can and continue on your
    beat.
  • D Approach the youths and calm the situation down
    by asking them to move on. Decide to deal with
    the issue of the can-throwing later on when the
    situation has calmed down.
  • E Make a mental note of who threw the can, radio
    for assistance and wait for back-up.

11
What does an SJE measure?
  • Job Knowledge
  • Organising for Results
  • Practical Intelligence
  • Commitment to high Standards
  • General Cognitive Ability
  • Call Centre Performance
  • Decision Making
  • Team work Knowledge
  • Negotiating/ Influencing
  • Aviation Pilot Judgement
  • Integrity
  • Conscientiousness, Emotional Stability
    Agreeableness

12
What does an SJE measure? Contd.
  • Best viewed as a measurement method and not a
    measure of a single construct (O Connell et al
    2007) outputs may be a single score or range of
    scores across constructs/ competencies
  • Can assess the integration of cognitive and
    affective based behaviours Lewis et al 2006,
    cited in Creighton Scott 2006.

13
Recent developments Principal Officer
  • Senior Grade within the Civil Service
  • Exercise to be used for existing Civil Servants
    and open candidates
  • Initial context setting brief with organisation
    chart to increase the fidelity of exercise
  • Candidates presented with 12 scenarios and
    provided with 5 possible actions asked to rate
    each in terms of their appropriateness on a 6 pt.
    scale from highly appropriate to highly
    inappropriate
  • 971 candidates sat the tests
  • 87.7 found it relevant or very relevant
  • 91.9 felt it was at the right level of
    difficulty
  • 82.4 felt it was fair or very fair

14
Considerations in the use of SJEs
  • Benefits
  • Effective assessment methods
  • Can balance out the potential adverse impact of
    cognitive tests
  • Flexibility Can measure a range of required
    constructs
  • High levels of face validity for candidates
  • Can be administered efficiently to large numbers
    of candidates
  • May be designed to replicate specialist jobs
    effectively useful realistic job preview

15
Considerations in the use of SJEs
  • Challenges
  • Challenges with Candidate feedback
  • Significant investment required to devise
    exercises
  • Balance needed when using exercises on a confined
    and open candidate pool
  • In confined competitions, a neutral context is
    needed rather than one favouring candidates from
    policy/ operational areas
  • Research suggests faking
  • Is easier when items are more transparent
  • Is more difficult when there is a heavier
    cognitive load
  • Is more likely with behavioural instructions
    when asked what would you do, people will respond
    with what they feel is best answer

16
Example Fire-fighter campaign
  • Online application only (over 4,300 applicants)
  • Comprehensive assessment process
  • Screening
  • Job relevant selection tests (video based)
  • Learning information
  • Understanding Information
  • Using numbers
  • Observation
  • Very positive feedback from candidates over 85
  • describing the screening process as clear,
    relevant fair
  • Follow up assessment
  • Interview
  • Team Exercise
  • Practical Orientation Exercise

17
(No Transcript)
18
Example Fire-fighter campaign - Video based tests
Learning Information During this test candidates
were presented with a series of short talks/
presentations on video during which they were
encouraged to take notes. The talks/
presentations were on fire-related topics.
Candidates were assessed on how much they had
learned about the topic through a series of
multiple choice questions. Observation This
test is about the ability to identify key
information presented on video (involving a
series of short films of objects set out in a
number of rooms. Within these rooms are a range
of items such as boxes, planks, containers
electrical equipment and furniture After the
video candidates were shown a plan of the rooms
and asked a number of multiple choice questions
about the video
19
Recent developments Emergency Medical
Controller
  • Emergency Medical Controllers take emergency
    medical calls and dispatch assistance
  • Job Analysis demonstrated requirement for
    attention to detail and very specific
    interpersonal skills when dealing with callers
    and other health care professionals
  • Sourced video based assessment
  • Call taker test replicated a call-centre.
    Candidates required to take notes, and after an
    excerpt from the phone calls, choose which of a
    series of options the call taker should follow
  • Dispatch test candidates had to dispatch
    emergency vehicles according to a set of
    priorities

20
(No Transcript)
21
Validity
  • Sourced from IPMA HR
  • Because of US legal context, very good validity
    evidence e.g
  • Correlation between call-taker test and academy
    performance was .30
  • Correlation with supervisors ratings was .20
  • Correlation with overall pass/fail on probation
    was .28
  • Number in sample 627

22
Case Study EO Disabilities Campaign 2007
  • Stage 1
  • On-line testing
  • Verbal Reasoning test
  • Skills and Experiences Questionnaire
  • Equality Monitoring Form
  • Stage 2
  • Paper and Pencil testing
  • Verbal Reasoning test
  • Numerical Reasoning test
  • Job Simulation Exercise
  • Stage 3
  • Competency-based interview and Work-sample test

23
Why Internet based testing?
  • Evolution from supervised computer based testing
  • Convenient for candidates
  • Applicant population generally familiar with
    technology
  • More efficient than face-to-face testing

24
Professional Concerns about Internet Based Testing
  • Significant debate about the validity of
    Internet Based Testing (Tippens et al, 2006)
    Key concern threat of cheating to the integrity
    and validity of ability tests
  • Concerns led Health Professions Council in
    South Africa to ban unsupervised Internet testing
    successfully challenged in the courts
  • Cizek (1999) Propensity to cheat is
    negatively related to job performance (-.30)
  • Burke and Wright (2008) SHL Verify range of
    tests
  • 578 applicants re-tested
  • Across the sample 5 flagged as aberrant about
    the rate expected by chance
  • Applicants to graduate programmes 9 on the
    numerical test, 18 on the verbal test

25
Case Study EO Disabilities Campaign 2007
  • No.
  • Applied 575
  • Sat Stage 1 on-line 311
  • Sat Stage 1 off-line 81
  • Passed Stage 1 281
  • Sat Stage 2 250
  • Passed Stage 2 124
  • Stage 3 October 2007
  • People currently being assigned to jobs

26
Stage 1 Feedback survey from candidates
  • Telephone survey conducted with over 100
    candidates 45 responded
  • 84 felt that doing the tests on line is a good
    idea for people with disabilities
  • 78 preferred to do the tests on-line vs in a
    supervised environment
  • more relaxed on-line
  • the test was much easier on a PC
  • 90 found it convenient to do the tests in their
    own time
  • 100 of candidates use computers everyday

27
References
  • Creighton, P Scott, N. (2006). An Introduction
    to Situational Judgement Inventories. Selection
    and Development Review 22, 3-6.
  • O Connell et al (2007). Incremental Validity of
    Situational Judgement Tests for Task and
    Contextual Job Performance. International Journal
    of Selection and Assessment. 15, 19-29.
  • McDaniel et al (2007) Situational Judgement
    Tests, Response Instructions and Validity A
    Meta-analysis. Personnel Psychology 60, 63-91.
  • Lievens, F., Peeters, H Schollaert, E. (In
    Press) Situational Judgment Tests A Review of
    Recent Research. Personnel Review

28
References
International Guidelines on Computer-Based and
Internet Delivered Testing International Test
commission (2005) www.intestcom.org On-line
Testing and the Disability Discrimination Act
(SHL) www.shl.com Cizek. G. J (1999). Cheating on
tests How to do it, detect it, and prevent it.
New Jersey LEA Tippins, N. T., Beaty, J.,
Drasgow, F.,, Gibson, W.M., Pearlman, K., Segall,
D.O, and Shepherd, W. (2006) Unproctored
Internet Based Testing in employment settings.
Personnel Psychology, 59, 189-225.
29
  • Thank You!

30
Chapter House, Abbey St, Dublin 1
aine.gray_at_publicjobs.ie
About PowerShow.com