Assessment in Education

1
Assessment in Education

• Patricia OSullivan
• Office of Educational Development
• UAMS

2

• Last Month Dr. Allen described a situation about
  an individual asking her how to get to Freeway
  Medical Center. How would you determine how
  successful Dr. Allen was?

3
Objectives

• Provide terminology and principles fundamental to
  assessment
• Introduce various assessment approaches
• Review tips for a common assessment Multiple
  Choice Questions
• Review elements for a less common assessment
  Portfolio

4
What is Assessment?

• Appraising performance through measurement to
  make an evaluation
• For what purpose?
• Formative
• Summative

5
What Assessments are Used at UAMS?
6
What Makes a Good Summative Assessment?

• Fair and practical
• Based on appropriate criteria that are shared
  with the learners
• Valid
• Reliable

7
Validity

• Key to the interpretability and relevance of
  scores
• Do the scores have a plausible meaning?
• What are the implications of interpreting the
  scores?
• What is the utility of the scores?
• What are the actual social consequences?

8
Validity Evidence for Proposed Interpretation

• Content/Construct Evidence
• Content matches the objectives teaching
• Score is shown to be meaningful
• Criterion/Predictor Evidence
• Score relates to future outcomes

9
Reliability

• Assessment data must be reproducible
• Written tests
• Internal consistency
• Cronbachs alpha or KR-20 (values range 0-1)
• Test-retest
• Correlation ranging from 0-1

10
Reliability (cont.)

• Rater-based assessments
• Interrater consistency or agreement (generally
  correlation coefficient or intraclass
  correlation)
• OSCE performance assessments
• Generalizability Theory

11
Improve Validity

• Match assessment with objectives
• Increase the sample of objectives and content on
  the assessment
• Review blueprint (or produce one)
• Use test methods appropriate to objectives
• Ensure test security

12
Improve Reliability

• Clear questions
• Appropriate time allotted
• Simple, clear and unambiguous instructions
• High quality scoring
• Increase number of observations or questions

13
Norm or Criterion Referenced

• Norm-referenced (relative standard)
• Use the results of the assessment to set the
  standards (e.g. proportion to receive each
  grade) performance judged by comparison with
  others
• Criterion-referenced (absolute standard)
• Learners achieve some minimal standard of
  competency

14
Standard Setting Procedures

• Judgments based on
• Holistic impressions of the exam or item pool
• Content of individual test items
• Examinees test performance

15
Anghoff Standard Setting Procedure

• Judges are instructed to think of a group of
  minimally acceptable persons
• For each item estimate the proportion of this
  group that will answer item correctly
• Sum up proportions to get the minimum passing
  score for a single judge
• Average across judges

16
Assessment Methods

• ACGME Toolbox
• Additional methods
• Other written assessments
• Essay
• Short answer and computation
• Patient management problems
• Modified essay questions
• Create a game
• Self-assessment

17
Matching Assessment Method and Objectives

• Examine ACGME matrix

18
Alternative Testing Modes

• Take-home tests
• Open-book tests
• Group exams
• Paired testing

19
Reducing Test Anxiety

• Make the first exam relatively easier
• Give more than one exam
• Give advice on how to study
• Encourage study groups
• Give a diagnostic test early

20
Reducing Test Anxiety

• Before a test explain format
• Provide sample items
• Provide a pool of final items with syllabus

21
Technology and Assessment

• Computer based testing
• Use of images and video clips
• Computer Adaptive Testing (CAT)

22
Take a Test
23
Multiple Choice Questions Writing Tips

• http//www.nbme.org/about/itemwriting.asp
• Simple suggestions
• Design for one best answer
• Place hand over stem. Can you guess the answer?
• Use a template
• Attend to Graveyard of Test Items

24
Template Examples

• A (patient description) is unable to (functional
  disability). Which of the following is most
  likely to have been injured?
• Following (procedure), a (patient description)
  develops (symptoms and signs). Laboratory
  findings show (findings). Which of the following
  is the most likely cause?

25
Graveyard

• B-type matching heading with words or phrases
  (can use heading more than once)
• D-type complex matchingtwo step process
• K-type stem with four options and choices are
• A1,2,3 only B1,3 only C2,4 only
• D4 only, Eall are correct

26
Portfolios
27
(No Transcript)
28
Portfolios

• Definition
• A purposeful collection of student work that
  exhibits to the student (and/or other) the
  students efforts, progress, or achievement in
  (a) given area(s).

29

• This collection must include student
  participation in
• Selection of portfolio content
• The criteria for selection
• The criteria for judging merit
• Evidence of student
• reflection

30
Portfolios

• Learning
• Showing progress
• Showing effort
• Showing remediation

• Assessment
• Showing competence

31
Presentations
Logs
Case Records
Evaluations
32
Criteria ______ ______ ______
33
Criteria
34
Portfolio
Reflection

• Why selected
• How reflects skills and abilities

35
Using Portfolios the Learner Engages in Assessment

• Making choices
• Critically self-examining
• Explaining what they are doing

36
Well-structured Portfolio

• Menu
• Selection criteria
• Scoring rubric
• Benchmarks
• Trained raters

37
Determine Reliability and Validity

• Reliability Generalizability theory
• Found needed 6 entries with 2 raters
• Validity number of studies
• Resident performance across years
• Resident performance in areas
• Resident perception
• Resident performance when program made an
  intervention
• Correlation of portfolio performance and other
  areas

38
10-Step Approach

1. Define the purpose
2. Determine competencies to be assessed
3. Select portfolio materials
4. Develop an evaluation system
5. Select and train examiners

39
10-Step Approach

1. Plan the examination process
2. Orient learners
3. Develop guidelines for decisions
4. Establish reliability and validity evidence
5. Design evaluation procedures