Quality Control in Survey Design: Evaluating a Rating Scale of Educators - PowerPoint PPT Presentation

Loading...

PPT – Quality Control in Survey Design: Evaluating a Rating Scale of Educators PowerPoint presentation | free to download - id: 71d4d5-NTM0O



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Quality Control in Survey Design: Evaluating a Rating Scale of Educators

Description:

Quality Control in Survey Design: Evaluating a Rating Scale of Educators Attitudes Toward Differentiated Compensation Shannon O. Sampson Kelly D. Bradley – PowerPoint PPT presentation

Number of Views:8
Avg rating:3.0/5.0
Slides: 27
Provided by: Unkno217
Learn more at: http://www.uky.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Quality Control in Survey Design: Evaluating a Rating Scale of Educators


1
Quality Control in Survey Design Evaluating a
Rating Scale of Educators Attitudes Toward
Differentiated Compensation
  • Shannon O. Sampson
  • Kelly D. Bradley
  • University of Kentucky

2
Background
  • Surveys- most common example of self-reported
    data collection and one of the most popular
    research methodologies for graduate studies and
    published papers in education (Aiken, 1988
    Babbie, 1992 Gay, 1981).
  • Even so, the efficiency and effectiveness of the
    instrument as a measurement tool is often
    overlooked or underemphasized.

3
  • Operationalizing and then measuring variables
    are two of the necessary first steps in the
    empirical research process. Statistical
    analysis, as a tool for investigating relations
    among the measures, then follows. Thus, the
    interpretation of analyses can only be as good as
    the quality of the measures. (Bond and Fox,
    2001)

4
Objectives of Study
  • Utilize Rasch analysis to evaluate the quality of
    a survey instrument designed to measure
    educators attitudes about differentiated
    compensation
  • Employ a data-driven model for improving survey
    instrumentation

5
Assumptions with traditional rating scale data
analysis
  • Each item contributes equally to the measure of
    the construct
  • Each item is measured on the same interval scale
  • Respondents have appropriately interpreted the
    directions
  • All items are written clearly such that only one
    interpretation is possible

6
However
  • Items generally represent different amounts of a
    variable
  • Scales are ordinal, so categories are not
    necessarily spaced equally
  • Respondents often misinterpret directions
  • Items are often open to multiple interpretations

7
Furthermore
  • Estimates for items depend on severity of
    respondents in sample
  • Estimates of item ratings cannot be compared
    across groups
  • Complete records required
  • Single standard error of measurement is produced
    for the composite of ratings

8
Rasch model
  • Probabilistic version of the scalogram
  • Parameters neither sample nor test dependent-
    missing data not problematic
  • Standard error estimates produced for each
    discrete raw score

9
Attitudes about differentiated compensation
  • Differentiated compensation Range of incentives
    added to present compensation
  • Salary bonuses for teaching in critical shortage
    areas
  • Financial support for seeking advanced degrees
  • Participation in voluntary career advancement
    opportunities

10
Instrumentation and Sample
  • 10 KY school districts involved in differentiated
    compensation program pilot
  • University of Kentucky faculty constructed a
    pencil and paper survey instrument
  • Survey administered to four groups
  • Teachers (n 438)
  • Mentor teacher- achievement coaches (n 60)
  • Principals (n 63)
  • Superintendents (n 10)

11
  • Prior to analysis, our preliminary ideas about
    items and persons we choose to study obligates us
    to form specific hypotheses about both items and
    persons.
  • (Wright Stone, 2004)

12
Evaluating the quality of the instrument
  1. Evaluate the coherence of the data (does a
    yardstick exist?)
  2. Evaluate the rating scale structure (how accurate
    is the yardstick?)
  3. Evaluate the individual items (can the yardstick
    be refined?)

13
1. Evaluate the coherence of the data (do I have
a yardstick?)
  • Have items been keyed as intended?
  • Are there problems with the data coding?
  • Are the items measuring only one variable?
  • Are all items pointing in the same direction?

14
Kentucky Department of Education Differentiated
Compensation Survey
15
2. Evaluate the rating scale structure (how
accurate is my yardstick?)
  • Do the mean measures for the responses in each
    category increase as the categories step up the
    scale in the direction defined as more?
  • Do the categories fit the expectations of the
    model?
  • How are the respondents using the rating scale?

16
Do the mean measures for the responses in each
category increase as the categories step up the
scale in the direction defined as more?
Empirical item-category measures for
administrators -2 -1 0 1 2
3 4 5 -----------------------
-------------------------- NUM ITEM
12 3 4
17 Non-cert Ts should pay
all... 1 23 4
7 linking teacher
salary to... 1 2 3
4 11 Students'
standardized tes... 1 2
3 4 2 DC
would not enhance the p...
1 2 34
18 When non-cert Ts are hired... 1
2 3 4
24 If Ts receive a DC bonus...deserve it
3 24
33 There is too much peer
pre... 1 2 3 4
25 The size of the
salary bon... 342
19 cert Ts
are more effective... 1 2
3 4 8
Ts receiving differentiate... 1
2 3 4
1 DC will attract better qua...
3 2 4
32 Ts believe their school stresses
excellence... 3 4
31 Ts feel
a sense of ownership in student lrng
3 4
28 Ts identify with their sch...
3 2 4
29 Ts take pride in being a
p... 3 4
30 Ts feel a sense
of ownership in school 3
4 27
Improving knowledge and sk...
3 4
34 Ts are encouraged to make suggestions
143
16 All Ts should be
required... ---------------------------------
---------------- NUM ITEM -2 -1 0
1 2 3 4 5 1
2 2423227367 34442512 212 1
PERSONS T S M S
T
17
How are the respondents using the rating scale?
Differentiated Compensation Survey- Administrators
CATEGORY PROBABILITIES MODES - Structure
measures at intersections P
--------------------------------------------
---------- R 1.0
O

B
A 111
B
.8 111
I 11
444 L
11
44 I 11
44 T .6
11 3333333333 44
Y 1 33
3333 44 .5 11
33 3344 O
1 33
44333 F .4 11
33 44 33
222222222 44
33 R 2222 1 222
44 333 E
222 33 1 222 44
33 S .2 222 33 11
22244 3 P 2222
333 111 444222
O 2 333 4
22222 N 3333333
4444444 111111 2222222 S
.0 444444444444444444
1111111111111 E
--------------------------------------------
---------- -3 -2 -1
0 1 2 3
PERSON MINUS ITEM MEASURE
Strongly disagree
Strongly agree
Agree
Disagree
18
Do the categories fit the expectations of the
model?
  • -------------------------------------------------
    -----------------
  • CATEGORY OBSERVEDOBSVD SAMPLEINFIT
    OUTFITSTRUCTURECATEGORY
  • LABEL SCORE COUNT AVRGE EXPECT MNSQ MNSQ
    MEASURE MEASURE
  • ---------------------------------------------
    ---------------
  • 1 1 93 4 -.76 -.85 1.12 1.39
    NONE ( -2.54) 1
  • 2 2 220 9 .32 .29 .97 .92
    -1.12 -.96 2
  • 3 3 982 40 1.31 1.36 1.00 .96
    -.67 .70 3
  • 4 4 1145 47 2.60 2.57 1.01 1.00
    1.79 ( 2.95) 4
  • ---------------------------------------------
    ---------------
  • MISSING 8 0 .78
  • -------------------------------------------------
    -----------------

Category fit is acceptable
Respondent use of scale is almost dichotomous
administrators
19
3. Evaluate the individual items (how can I
refine my yardstick?)
  • Do the items fall into the hypothesized
    hierarchy?
  • Do the items spread evenly across the intended
    range of the instrument?
  • Do any items clump at a point on the scale?
  • Which items are misfitting? Why might they be
    misfitting?

20
Do the items fall into the hypothesized
hierarchy?
  • Students standardized test scores would improve
    if a differentiated compensation program were
    adopted
  • Differentiated compensation would not enhance the
    positive relationship among teachers and
    administrators
  • A differentiated compensation program will
    positively affect teacher morale
  • Relations between administrative and
    instructional staff will be negatively affected
    if a differentiated compensation program is
    adopted
  • Differentiated compensation will have a negative
    impact on the morale of the teachers in the
    system
  • Differentiated compensation programs recognize
    teacher contributions to student learning
  • Differentiated compensation programs help recruit
    teachers who can improve student learning
  • A differentiated compensation program will result
    in a higher teacher retention rate
  • It is appropriate for teachers to receive bonuses
    to serve in rural schools classified as a
    difficult assignment or hard-to-fill position
  • The size of the salary bonus I could receive to
    become certified in a critical shortage area or
    difficult assignments must be large enough to
    motivate me.
  • A differentiated compensation program helps
    recruit better-qualified people to the teaching
    profession
  • A differentiated compensation program helps
    retain teachers in critical shortage areas
  • I identify with this school
  • All teachers should be required to be certified
    to teach
  • I take pride in being a part of this school
  • Improving teachers knowledge and skills enhances
    student learning
  • School districts should support teachers who are
    voluntarily advancing their careers
  • I feel a sense of ownership in student learning


  • (Teacher hierarchy)

Difficult to endorse
Easy to endorse
21
Do the items spread evenly across the intended
range of the instrument?
MAP OF PERSONS AND ITEMS MEASURE

MEASURE ltmoregt ---------------------
PERSONS -- ITEMS --------------------- ltraregt
5
5



4

4



X 3
X
3 T X
XX T X
X
XXX
X S 2
XXXXX X 2
XXXXX
XXXX
XXXX
XXXX MS
XXXXXX XX 1
XXXXXXXXX XX 1
XXXX XX
XXXXX S XXX
X XXXX
XX X
X 0
X TM X
0
XXXX
X X X

-1

-1
X S
X X

-2
XXX
-2
XXX

T
-3

-3



-4
-4
ltlessgt --------------------- PERSONS -- ITEMS
------------------ltfrequentgt
22
Which items are misfitting? Why might they be
misfitting?
Infit ZSTD Outfit ZSTD Item Misfits for Teachers
7.2 9.6 There is too much peer pressure here to do a good job
4.9 5.8 All teachers should be required to be certified to teach
5.9 7.3 Certified teachers are more effective than non-certified teachers
7.3 8.6 Non-certified teachers should pay all the costs of becoming certified
3.1 3.9 This school stresses excellence
3.7 3.4 Im encouraged to make suggestions about how we can be more effective
Do these items tap the construct?
23
Infit ZSTD Outfit ZSTD Item Misfits for Teachers
5.3 5.6 School districts should pay for university coursework in content areas
4.8 5.4 Linking teacher salary to student achievement on standardized tests has no place in education
2.9 4.2 When non-certified teachers are hired, school districts should pay all the costs of becoming certified
3.0 3.4 The size of the salary bonus I could receive to become certified in a critical shortage area or difficult assignments must be large enough to motivate me
These items may have multiple interpretations In
which content areas? On which standardized
tests? For all teachers and all content
areas? What is large enough to motivate?
24
Infit ZSTD Outfit ZSTD Item Misfits for Teachers
-5.2 -5.1 Differentiated compensation will positively affect teacher morale
-4.7 -5.3 Differentiated compensation provides incentives for growth
-7.6 -7.3 Students standardized test scores would improve in the district if a differentiated compensation program were adopted
-6.6 -6.4 Differentiated compensation programs help recruit teachers who can improve student learning
Less variability than the model would
predict Redundancy or High agreement among
respondents
25
  • The problem of measurement, and especially of
    attaining interval scales, is an extremely
    serious one for the social and behavioral
    sciences. It is unfortunate that in their search
    for quantitative methods, researchers sometimes
    overlook the question of level of measurement and
    tend to read quite unjustified meanings into
    their results However, the core problem of level
    of measurement lies outside the province of
    mathematics and statistics.
  • (Hays, 1988)

26
Educational Importance
  • The education community will benefit by receiving
    better-informed results by collecting data using
    a more valid and reliable instrument.
  • Offers a sound methodology for evaluating the
    quality of the measurement instrument
About PowerShow.com