Checklists

1
Checklists Rating Scales

• Scoring Performances

2
Goals

• understand the importance of observing
  performances and grade them appropriately
• understand, spot, compensate for usual errors
• understand and use appropriately the following
  observational tools
• checklists
• rating scales
• exemplars
• keys

3
(No Transcript)
4
Principles of Effective Rating

• Characteristics should be educationally
  significant in harmony with departments vision,
  goals, and objectives and desired learning
  outcomes.
• Characteristics should be directly observable
  occur in the setting and clearly visible to an
  observer skills less so are interests,
  feelings, and attitudes.

5
Principles of Effective Rating

• Characteristics and points on the scale should be
  clearly defined.
• Between three and seven rating positions should
  be provided, and raters should be permitted to
  mark at intermediate points.
• Raters should be instructed to omit ratings when
  they feel unqualified to judge.

6
Principles of Effective Rating

• Ratings from several observers should be combined
  whenever possible -gt greater reliability.

7
Checklists and Rating Scales

• are MONITORING and
• RECORD-KEEPING devices
• With increased emphasis on self-reflection and
  self-assessment, individual students should use
  self-checklists and rating scales to ASSESS THEIR
  OWN PROGRESS and DEVELOP AN IMPROVEMENT PLAN.

8
Effective Observations

• from adult framework
• recorded
• objective nonjudgmental
• consider whole learner
• regular, systematic, frequent observing
• include spontaneous ongoing observations

• efficient recording
• purposeful
• monitor product process before, during, after
  task
• from many points of view
• use spectator and participant modes

9
Affective Measurement Problems

• Fakability
• Self-deception
• Semantic problems
• Criterion inadequacy

10
Checklists Rating Scales

• Checklist Attitude Yes or No
• Checklist Attitude Yes or No?
• shows originality Yes No
• respects own work Yes No
• is easily motivated Yes No
• Rating Scale Attitude -- Degree
• Rating Attitude SA A U D SD
• shows originality SA A U D SD
• respects own work SA A U D SD
• is easily motivated SA A U D SD

11
Constructing Checklists

• Base on course objectives.
• State in clear and precise terms.
• Avoid generalities focus on specifics.
• Design them so they can be used as a basis for
  discussion by instructor and student. Thus,
  write them so they are easily understood by the
  student.
• May also be used for self- and peer-evaluation.

12
Constructing Checklists

• Keep them short so that they can focus on a few
  specifics.
• May be designed for either a single evaluation or
  a cumulative record.
• Can be used for diagnostic, formative, and/or
  summative evaluation as well as a basis for
  anecdotal comment, as well as an instructional
  tool for learner to follow in practicing the
  skill.

13
Differentiating Assessment Tools

• CHECKLISTS
• psychomotor social skills
• indicate presence or absence/mastery or
  non-mastery of skill
• may indicate sequence of actions in a performance

• RATING SCALES
• for psychomotor and affective skills
• indicate quality, degree, or frequency
• of skill or trait
• useful in summarizing general impressions

14
Checklist Preparation for Varnishing

• __ sands and prepares surface properly
• __ wipes dust from surface with appropriate cloth
• __ selects appropriate brush
• __ selects varnish and checks varnish flow
• __ pours needed amount of varnish into clean
  container, etc.

15
Rating Scale Job Attitude

• is enthusiastic G Av N Imp
• is willing to work cooperate G Av N Imp
• desires to improve G Av N Imp
• is a good team worker G Av N Imp
• is concerned with safety G Av N Imp
• works without supervision G Av N Imp
• aware of job requirements G Av N Imp

16
T-Form Encouragement

• Looks Like__________Sounds Like_____

17
Why Use Observation Checklists

• A quick and easy way to observe and record many
  of the skills and behaviors rarely assessed prior
  to final test or summative exam.
• Show areas that need work early enough to provide
  help and correct before student fails area/unit.

18
Why Use Observation Checklists

• Provide opportunities to change gears re
  feedback.
• The characteristics of intelligent behavior can
  be identified, taught, and checked.
• Observation checklists let us check off whether
  student can demonstrate skill or attribute being
  measured. Develop base practice check progress.

19
Why Use Observation Checklists

• They focus on observable performances or criteria
  that are often more especially useful for
  monitoring growth or need for improvement. They
  are more meaningful or authentic.
• Useful re what learners can do and what their
  learning needs are.

20
Why Use Observation Checklists

• Useful to discover patterns, assess progress, and
  make plans to help students continue their
  learning.

21
How to Use Observation Checklists

• First, conduct a TASK ANALYSIS to determine
  specific areas to include in the observation
  checklist.
• Then, ensure students are aware of the areas that
  will be observed.
• Do this by training participants in what the
  skill LOOKS LIKE and SOUNDS LIKE even
  FEELS LIKE.

22
How to Use Observation Checklists

• Students may be assessed by peers or perform a
  self-assessment, before an instructor assessment.
• Model, Teach, and Practice the skills prior to
  observation.
• Students should work with the instructor to
  develop the indicators.

23
Indicators for Persistence

• knows how to access information
• tries several approaches
• does not give up quickly
• has patience
• brainstorms alternative solutions
• checks own work
• Do so in a T-Form.

24
Making Valid Observations

• Plan in advance what is to be observed and
  prepare an observation list, guide, or form to
  make the observations more objective and
  systematic.
• Concentrate on a few significant behaviors at a
  time.
• Use clear, unambiguous terminology and
  accompanying descriptions in the observation
  tools.

25
Making Valid Observations

• Each trait item should be mutually exclusive
  for coding.
• The observer must be cognizant of the
  time-sampling errors use frequent, short
  observations distributed over a period of several
  weeks and at different times of the day and week.

26
Making Valid Observations

• Coordinate the observations with your teaching
  instructional strategies and observations must be
  coordinated.
• Extensive observations should be selective the
  most needy cases.
• Carefully record and summarize the observation
  immediately after it has occurred.

27
Making Valid Observations

• Make no interpretations concerning the behavior
  until later on. Doing so may interfere with
  objectivity.
• Have categories and coding schemes that are
  simple to use, that call for behaviors easily
  observed, and that deal with behaviors that can
  be conveniently recorded. Inferences should
  occur only after data have been gathered and
  recorded.

28
Making Valid Observations

• Wherever possible, observers and observations
  should be unobtrusive. When people know they are
  being observed, they often tend to avoid or
  overemphasize certain behaviors to gain attention.

29
Checklists Basics

• for psychomotor processes or products and for
  personal/social growth
• for both processes and products
• for analyzing sequences and correct order of
  actions
• for traits present in a product

30
Checklists Basics

• Limitations
• indicates present or absent only no degree
  specified an all-or-nothing scale.
• Not useful for summarizing impressions as no
  degree indicated.

31
Checklists Basics

• Perform a careful task analysis to determine
  component behaviors.
• Clearly specify the actions/traits to be
  observed.
• Include in the list common errors where they are
  likely to occur.
• Arrange procedures in the order they are expected
  to occur.

32
Checklists Basics

• Keep separate checklists for each person.
• Transfer results to a master list of all
  candidates for making comparisons.
• Give clear directions.
• Train observers.

33
Common Errors in Rating

• Personal Bias general tendency to rate all at
  approximately the same position on the scale
• generosity error - rates all at high end
• severity error rates all at low end
• central tendency error rates all as average

34
Common Errors in Rating

• Halo Effect our general impression of a person
  influences the rating of individual
  characteristics
• Logical Error two characteristics intelligence
  and achievement gifted and poor social
  adjustment are rated as more or less related
  than they actually are.

35
Common Errors in Rating

• Raters Attitude doesnt view rating as
  important and doesnt put much into it.
• Not enough time to do a good job need more
  extensive opportunities for observation.

36
Resulting Problems

• Puts in doubt a single rating of an individual
  Limits the range of an individuals ratings.
• Obscures strengths and weaknesses on different
  traits because the individual receives similar
  ratings on all characteristics.
• Higher relationship implied than actually exists.
• Not done, invalid, and/or flawed results.

37
Overcoming Problems

• Proper design, training, and use of the scales.

38
Performance Indicators

• Quality
• Quantity
• Waste
• Safety
• Improvement
• Focus Finished student product
• Process Process student uses to produce product

39
Quality Indicators

• conformity to overall specifications
• dimensions, spacing, and/or strength
• suitability for intended purpose
• general appearance
• error rate during performance
• efficiency of steps
• choice of tools, equipment, and/or materials

40
Time Indicators

• time needed to complete product or perform
  process
• number of products produced

41
Improvement Indicators

• improvement in one or more qualitative
  characteristics
• improvement in one or more quantitative
  characteristics
• reduction in the number of steps for which
  assistance is required

42
Waste Indicators

• number of unacceptable products produced
• amount of excess material used during performance

43
Safety Indicators

• Degree to which completed product is safe for
  intended use
• Safe handling of tools, materials, and/or
  equipment
• Accident rate during performance

44
What is QUALITY?

• If something occurred occasionally, it would
  occur __ of the time.
• If something occurred sometimes, it would occur
  __ of the time
• If something occurred frequently, it would occur
  __ of the time
• If something occurred usually, it would occur __
  of the time
• If something occurred often, it would occur __
  of the time

45
Rating Scale Basics

• Use for psychomotor and affective domains
• give quality, frequency, degree, or level
  information
• for processes or products

46
Rating Scale Limitations

• Subject to a number of biases leniency,
  strictness, modesty, halo effect.
• Subject to social expectations and, hence, to
  faking and self-deception.
• Subject to interpretation regarding semantic
  problems.
• Subject to criterion inadequacy.

47
Precautions/Considerations

• Begin with a blueprint of the behaviors to ensure
  valid sampling within the area.
• Clearly define traits in most behavioral way
  possible.
• Divide the rating continuum into as many points
  as needed for clarity usually between three and
  seven.

48
Precautions/Considerations

• Clearly define the anchors on the continuum so
  there is no question as to what each rating
  means.
• Train and motivate the raters to be as accurate
  and objective as possible.

49
Sample Rating Scale Degree of Confidence

• 1 little or no 2 some 3 great confidence
• inquiry teaching 1 2 3
• instructional modules 1 2 3
• simulations 1 2 3
• lecture 1 2 3
• leading a discussion 1 2 3
• questioning 1 2 3

50
Defining Steps on a Scale

• Numeric Anchors numbers, often accompanied by
  verbal cues
• Degree of Agreement Anchors Completely Disagree
  to Completely Agree Never to Always
• Adjectival Anchors Bipolar Adjectives define
  endpoints with numbers in between
• valuable 7 6 5 4 3 2 1 worthless
• passive 1 2 3 4 5 6 7 active