Title Slide - PowerPoint PPT Presentation

1 / 107
About This Presentation
Title:

Title Slide

Description:

Vice President, Solution Services. Prometric. The Past, Present and Future. Why ... is compared to the proper chronological order, with each object receiving a ... – PowerPoint PPT presentation

Number of Views:75
Avg rating:3.0/5.0
Slides: 108
Provided by: CREI152
Category:

less

Transcript and Presenter's Notes

Title: Title Slide


1
The Future of Testing How You Can Take
Advantage of It, Today!
David MeissnerVice President, Solution
ServicesPrometric
April 23, 2008
2
Introduction
  • The Past, Present and Future
  • Why Do We Test?
  • Why Technology-Based Testing?
  • Choosing A Delivery Option
  • Computer-Based Testing
  • Internet-Based Testing
  • Conversion Strategies
  • Choosing a Vendor
  • Recent Innovations

3
Testing and AssessmentWhere Have We BeenAnd
Where Are We Going?
4
  • Prediction is very difficult,
  • especially about the future.
  • Niels Bohr (also attributed to Mark Twain)

Difficult to see, always in motion is the
future Yoda, (also attributed to George Lucas)
5
A Very Brief History of Testing
  • 2200 B.C. Pre-Employment Testing

206 B.C. Psychological Testing
1855 British Civil Service Exams
1917-1945 U.S. Army Testing
1950s U.S. Armed Forces Testing
1960s Widespread K-12 Testing
1968 ASVAB
1970s Mainframe-based CBT
1980s Computer-AdaptiveTests
  • 1991 PC-based CBT
  • 1993 Performance-based CBT
  • 1995 Computer Simulations
  • Video-based CBT
  • 2005 Computer Mastery Testing
  • 2007 Emulation Testing
  • 2008 Dynamic Forms Generation

6
Delivery Technologies - 1993
  • No Data Transfer Standards
  • Cumbersome Localization Process
  • Heavy Reliance on Call Center Operations
  • Client Server and Mainframe Delivery
  • Low Resolution Graphics
  • Relatively Simple Item Types
  • Limited Data Collection

7
Computer-Based Testing - 1993
8
Internet-Based Testing - 1993
9
Delivery Technologies - 2008
  • Vendor-Defined Data Transfer Standards
  • Global Availability and Consistency
  • Internet-Enabled Services
  • Platform Independent Delivery
  • High Resolution Graphics and Multimedia
  • Robust Item Types
  • Rich Data Collection

10
Computer-Based Testing - 2008
11
Internet-Based Testing - 2008
12
Delivery Technologies - 2013
  • Independent Data Transfer Standards
  • Universal, Multi-Language Exams
  • Just-In-Time Delivery
  • Internet-Enabled, Personalized Testing
  • Immersive and Multimedia Rich
  • Performance-Based, Real Time Assessments
  • Seamless Integration With the Real-World

13
Why Do We Test?
  • To Meet Strategic Objectives
  • To Gather Information
  • To Maintain Professional Standards
  • To Measure Knowledge, Skills and Abilities
  • To Certify, License or Credential Individuals
  • To

14
Continuous Learning
Content Development
Psychometric Oversight
Item / Test Performance
Training Materials
Learning Performance
Test Material
Analytics Validity, Reliability, Fairness,
Effectiveness
Testing
Training
Shared Resources
Shared Resources
15
Comprehensive Testing System
16
Test Development Services
  • Job Analysis
  • Identifies what to measure
  • Item Development
  • Creation and development of test items
  • Test Design Construction
  • Development of content and structure
  • Psychometric Analysis
  • Evaluates validity, reliability and fairness

Confidential
16
17
Test Delivery Services
  • Test Publishing
  • Format, review, and package for distribution
  • System Management
  • Project, account and program management
  • Registration Scheduling
  • Authorization, authentication event scheduling
  • Test Administration
  • Test delivery methodologies and models
  • Results Processing
  • Test scoring and reporting

Confidential
17
18
Data Management Services
  • Data Storage
  • Application service provider model
  • Data Administration
  • Data scrubbing maintenance
  • Data Reporting
  • Reporting to support programs and organizational
    needs analysis
  • Data Integration
  • 3rd party software support

Confidential
18
19
Why Technology Based Testing?
  • To Increase
  • Immediacy
  • Convenience
  • Efficiency
  • Efficacy
  • Security
  • Face Validity
  • Test Taker Satisfaction

20
Immediacy
  • Faster Time to Market
  • Shorter Test Development Cycle
  • Availability of Immediate Scoring Test-Taker
    Feedback
  • Nightly Data Feeds To Speed Analysis

21
Convenience
  • Tests Available When and Where Test-Takers Choose
  • Availability of On-Demand Scheduling and
    Registration
  • Test Results Collected and Reported
    Electronically
  • Test-Taker Access to Testing History and Other
    Information

22
Efficiency
  • Minimized Item Exposure Risk
  • Optimized Item Pool Utilization
  • Potential for Year-Round Test Delivery
  • Normalized Staffing Demand
  • Leveraged Operating Costs

23
Efficacy
  • Highly Effective Test Delivery Model
  • Proven Comparability With Paper and Pencil
  • Engaging Item Types
  • Real-World, Simulations and Performance-Based
    Content
  • Data-Rich Test Results

24
Security
  • Paper-Less Delivery
  • Streamlined Logistics
  • Video/Audio Session Monitoring and Recording
  • Effective Item Navigation
  • Data Analysis and Forensics
  • Fragmentation and Encryption of Data
  • Emerging Biometric Capabilities
  • Full Session Keystroke Capture

25
Face Validity
  • Reference Materials and Exhibits
  • Performance-Based Items
  • Simulations
  • Live Application Testing

26
Test-Taker Satisfaction
  • More Frequent Testing
  • More Testing Locations
  • Potentially Shorter Testing Times
  • Immediate Scoring and Performance Feedback
  • Engaging Item Types and Real-World Testing

27
Test Delivery OptionsChoosing The Solution
Thats Right For You
28
When To Choose IBT
  • Highly Dispersed Testing Population
  • Relatively Small Candidate Base (lt1,000)
  • Readily Available Proctors
  • Very Frequent Content Changes
  • Less Rigid Delivery Conditions
  • Lower Test Complexity
  • Low to Medium Stakes Security
  • Need for Lower Test Delivery Price

29
Selected IBT Attributes
  • Immediacy
  • Ease of Use
  • Lower Cost
  • 24X7 Availability
  • Scalable Systems
  • ASP Hosting
  • E-commerce Support
  • Immediate Scoring and Reporting Availability
  • Hardware Independent
  • Powerful Authoring Capability
  • Prompt Editing Deployment
  • Eligibility Processing
  • Detailed Reporting
  • Full Recovery Capability
  • Multi-Language Support
  • Proctor Management

30
Thin versus Thick
  • Thin Client Delivery
  • No Software Installation or Content Download
  • Platform Independent
  • True anytime, anywhere Delivery
  • Thick Client Delivery
  • Locally Installed Software and/or Test Content
  • Operating System Dependency
  • Requires Advance Planning and Preparation

31
When To Choose CBT
  • Evenly Distributed Testing Population
  • Modest to Large Candidate Base (gt1,000)
  • Need for Impartial Proctoring
  • Relatively Stable Content
  • Desire for Consistency and Precision
  • Flexibility to Address any Content
  • Mid to High Security Requirements
  • Interest in Outsourced Administration

32
Selected CBT Attributes
  • Psychometric Rigor
  • Advanced Item Types
  • Performance-Based Testing
  • Integrated Multimedia
  • Custom User Interface
  • Extensive Physical and Data Security
  • Extensibility
  • Legal Defensibility
  • Immediate Scoring and Reporting Availability
  • Highly Scalable
  • Precise Item Section Timing
  • Exhaustive Data Collection
  • Reliable/Consistent Delivery
  • Professional Test Center Staff
  • Technical Support
  • Full Recovery Capability
  • Modular (customizable) Systems
  • Multi-Language Support

33
Test Delivery OptionsBricks and MortarClient
Server Based Solution
34
Delivery Models
  • Linear Testing
  • Multiple fixed-length exam forms
  • Random or sequential delivery (items, groups,
    sections)
  • Static and Dynamic Branching
  • Linear on the Fly Testing (LOFT
  • Dynamic exam form generation
  • Computer Mastery Testing (CMT)
  • Variable length testing with pre-defined testlets

35
Item Types and Associated Materials
36
  • Supports single response and multiple response
    multiple choice items
  • Item responses may have both positive and
    negative score values
  • Either Classical Test Theory or Item Response
    Theory Statistics can be generated in real-time

37
  • Static images (photographs and illustrations)
    may be associated with stems and individual
    answer options

38
  • A variety of supplemental material and
    reference documents can be made available to the
    candidate through the use of one or more exhibit
    buttons
  • As with all the design elements of the user
    interface the labels can be customized to meet
    specific needs

39
  • The size and placement of the exhibit materials
    is predefined during the publishing process to
    optimize screen real estate and ensure a
    consistent candidate experience

40
  • Scenario-based exams, in which a reading
    passage is integral to the presentation of the
    items, benefit from a split screen in which the
    passage and the item(s) are shown on the screen
    simultaneously

41
  • A vertical orientation often reduces the amount
    of scrolling required for scenario-based items,
    particularly when multiple items are associated
    with a single passage

42
  • Extensive reference materials and multi-part
    bodies of knowledge can be invoked through a
    single exhibit button

43
  • This document viewer allows access to multiple
    reference items
  • Various tools allow candidates to easily
    navigate dozens, and in some cases, hundreds of
    pages of text.

44
  • One or more video clips can be embedded into an
    item
  • The controls available to the candidate and the
    appearance of the media player can be fully
    customized

45
  • A standard or scientific calculator can be
    embedded into an item

46
  • To maximize the space available for other
    elements of the item the calculator can be
    invoked through a dedicated calculator button

47
  • The size and placement of the calculator is
    pre-defined to avoid overlapping important
    information

48
  • Audio clips may be made available to candidates
    within the APC testing center network, where all
    testing workstations are equipped with headphones

49
  • High resolution animations can be used to
    simulate real-world activities within a
    performance-based exam

50
  • Essays and other constructed response content
    can be gathered with the aide of integrated word
    processing capabilities

51
  • A very effective alternative to multiple choice
    can be the drop down item type

52
  • A very large number of potential responses can
    be included in the drop down list, reducing the
    potential of guessing and memorization

53
  • Hot-Area or Point and Click items utilize
    images (e.g. photographs, drawings, documents) as
    the means of response capture

54
  • Candidates point to the correct response(s)
    with their mouse and click on their selection(s)
  • The software then records the coordinates of
    the selection(s) and scores the item

55
  • Drag-and-Drop is another alternative item type
    that requires candidates to discern the correct
    answer, or sequence of answers, to the task that
    has been assigned

56
  • Candidates point to an object they wish to move
    and use their mouse to drag the object to the
    appropriate location

57
  • This particular item requires the wars on the
    left to be placed into the proper sequence on the
    right

58
  • Once the sequence is complete it is compared to
    the proper chronological order, with each object
    receiving a positive or negative score based on
    its relative placement

59
  • This item type is often used to measure
    matching and identification skills and can be a
    very efficient assessment tool

60
  • When taking paper and pencil exams many
    candidates mark their test booklets to indicate
    responses they believe are incorrect

61
  • A comparable function, known as strikethrough,
    allows candidates to insert visual reminders to
    themselves of answer options they believe are
    incorrect
  • Strikethrough is invoked by simply clicking on
    the text of an answer option

62
  • To facilitate recall when using extensive
    reference materials many candidates are
    accustomed to highlighting relevant pieces of text

63
  • The ability to highlight text within a scenario
    or passage can be enabled within the driver to
    ensure that the testing process closely
    represents real-world activities
  • Highlighting can be invoked by simply dragging
    the cursor over the desired text

64
  • Unicode compliance means that exams can be
    delivered in any language (single or double byte)
  • Exams are currently being delivered in more
    than 15 languages

65
  • Test sponsors may choose to make a review
    screen available throughout the exam to
    facilitate candidate navigation and provide a
    quick status check on their progress

66
  • Immediate score reporting can be provided to
    the candidate, both on screen and in printed form
  • Reports can be customized to present text and
    graphical representations of performance, and
    where possible, may include a digital photograph
    of the candidate

67
Test Delivery OptionsAnytime,
AnywhereInternet Based Solution
68
Introduction
  • Comprehensive Internet-Based testing platform
  • Item Authoring
  • Test construction
  • Proctor/Invigilator Management
  • Candidate Registration
  • eCommerce Payment
  • Test Delivery
  • Reporting and Data Management

69
System Overview
  • Secure Thin-Client Architecture
  • Centrally hosted solution from load-balanced
    servers
  • No software or exam content downloads required
  • Operating system and browser independent
  • Ability to deliver within firewall-protected
    environments
  • Full session recovery and backup capability
  • Global content management and version control

70
Item Types and Associated Materials
71
  • Shown here is the standard item layout for a
    single-response, multiple-choice item

72
  • A full-function, online calculator can be made
    available when authoring an item that might
    benefit from this tool

73
  • Essays and other writing assignments may be
    developed using the integrated word processor

74
  • Test authors may choose to display an onscreen
    score report or to withhold scores until a later
    date
  • Onscreen score reports provide a test a graphic
    representation of performance
  • Item-level feedback may also be provided

75
  • Authors are provided with an extensive tools to
    manage the development and ongoing maintenance of
    exams

76
  • All of the exams for which an author is
    authorized to develop/maintain are shown on the
    test inventory
  • From this screen existing exams may be modified
    or new exams developed

77
  • Item authoring can be accomplished by either
    simple data entry or the importation of content
    from another application
  • A complete user manual is continuously
    accessible from the ? symbol in the upper right
    hand corner of the interface

78
  • The system offers several delivery styles
  • Complete randomization
  • Randomization within sections
  • Specified sequence within sections
  • Specified sequence through the test

79
  • The frequency with which a candidate may take
    an exam is customizable to more effectively
    manage item exposure

80
  • A wide variety of scoring models can be applied
    to an exam to meet specific measurement criteria

81
  • Test section performance can be scored and
    rules established for their impact on overall
    test scores

82
  • Test authors can integrate a wide variety of
    multimedia elements into an exam, including
    static graphics, audio, video, animations and
    simulations

83
  • An extensive array of item types are supported
    by the system, providing significant flexibility
    to item authors and test developers

84
  • Item authoring involves the same data entry
    process used in the creation of test design and
    specifications

85
  • Context sensitive help is available for all of
    the fields marked with a ? symbol, providing
    additional technical detail regarding the
    particular feature

86
  • To maximize the available space for different
    content components the system offers four
    templates that can present items in either a
    vertical or horizontal format

87
  • Item authors can easily access graphics and
    other multimedia components that were previously
    uploaded into the media manager and then embed
    them within item stems, answer options and
    reference materials

88
  • To ensure proper placement of the various design
    elements authors may preview items prior to
    publication

89
  • Once the exam is complete authors may
    immediately place the test into production or
    into one of several off-line states. This allows
    subsequent authors and/or editors to access the
    exam prior to publication

90
  • An extensive array of more than 24 psychometric,
    statistical, and management reports are
    available to the author

91
  • Administrators are provided with a series of
    tools to facilitate the effective management of
    exams and candidates

92
  • To ensure data-driven decision making
    administrators have access to a variety of
    reports at a technical and business level

93
Converting Paper and Pencil To Computerized
Delivery
94
Conversion Strategies
  • Keep Things Simple
  • Ease candidates into CBT
  • Utilize existing test forms and item types
  • Maintain current testing periods
  • The Big Bang
  • Concentrate changes into one big event
  • Leverage advanced CBT functionality
  • Reengineer testing schedule

95
Test Development Steps
  • Decide on a testing model
  • Determine development strategy
  • Evaluate existing item bank
  • Pre-test new items
  • Analyze performance
  • Select items for initial final form(s)
  • Calibrate items
  • Assemble initial final form(s)

96
Test Development Steps (cont.)
  • Equate final form(s) to a previous exam form or
    set new cut score
  • Attach new pre-test item packets to initial final
    forms (20 of total items)
  • Deliver final forms pre-test item packets
  • Periodically repeat the above analysis process on
    pre-test items and gradually create new final
    forms
  • Repeat until initial item pool is pretested

97
Hypothetical Example Development Requirements
  • 100-item form administered currently
  • 2,400 exams to delivered per month, on demand
  • Pool of 1,000 items exists, but no pre-test data
    have yet been gathered
  • No new practice analysis or item development
    needed
  • Cut score to be carried over from a previous exam
    form
  • One-parameter (Rasch) measurement model to be used

98
Hypothetical Example Initial Fixed Form CBT
  • Gather 300 item responses on a live PP form
  • Calibrate PP items and establish cut score
  • Create an initial oversized beta form for CBT
    delivery (150 items) with 20 of items
    overlapping with PP form.
  • Deliver beta form to 300 candidates
  • Use beta data to evaluate, select, and calibrate
    items.

99
Hypothetical Example Initial Fixed Form CBT
(cont.)
  • Assemble items into a final 100-item final form
  • Establish cut score by equating with PP form.
  • Create six 25-item packets of pretest items
    (Packets 1-6)
  • Deliver CBT Form ABeta Packets (A1-A6) to live
    candidates. Score candidates on the final form
    items but not the beta items.
  • When 300 responses are collected on each beta
    packet, repeat the cycle of item analysis,
    calibration, forms assembly, and beta packet
    creation

100
Migration Issues To Consider
  • Is the CBT to be equivalent to an existing PP
    Exam?
  • Are there additional competencies that should be
    included?
  • What new item types are being considered?
  • Drop-Down Selection
  • Point and Click
  • Drag and Drop
  • Constructed Response

101
Migration Issues To Consider (cont.)
  • What delivery model is being considered?
  • Linear, Linear on the Fly, Computerized Mastery,
    etc.
  • Will new development be required before the
    computerization phase?
  • Will tests be offered continuously or in windows?
  • How much pressure will be placed on the test item
    pool?
  • How often must the item pool be refreshed for
    technical reasons?
  • How many forms will be required?

102
Choosing A Vendor
103
Assess Your Needs And Expectations
  • Define the program objectives
  • Validate the anticipated stakeholder benefits
  • Determine your budget and timeline
  • Identify your internal team
  • Perform a gap analysis
  • Inventory outsource requirements

104
Types of Service Providers
  • Implementation Consultants
  • Test Development
  • Psychometrics
  • Test Delivery
  • Data Management
  • Security
  • Marketing

105
Evaluation Criteria
  • Relevant Experience
  • Organizational Expertise
  • Size and Stability
  • Technical Capability
  • References
  • Consultative Approach
  • Innovation
  • Price

106
Potential Sources of Information
  • Network with colleagues
  • Association of Test Publishers
  • www.testpublishers.org
  • National Organization for Competency Assurance
  • www.noca.org
  • Council on Licensure, Enforcement Regulation
  • www.clearhq.org

107
Questions And Ideas
David Meissner david.meissner_at_prometric.com
Write a Comment
User Comments (0)
About PowerShow.com