Educational Research: Instruments caveat emptor

1
Educational Research Instruments (caveat
emptor)
EDU 8603 Educational Research Richard M. Jacobs,
OSA, Ph.D.
2
Instruments

• tools researchers use to collect data for
  research studies (alternatively called tests)

3
The types of instruments
1. Cognitive Instruments
2. Affective Instruments
3. Projective Instruments
4
1. Cognitive instruments...

• Measure an individuals attainment in academic
  areas typically used to diagnose strengths and
  weaknesses

5
Types of cognitive instruments...

• achievement tests

provide information about how well the test
takers have learned what they have been taught in
school
achievement is determined by comparing it to the
norm, the performance of a national group of
similar students who have taken the same test
6

• aptitude tests

measure the intellect and abilities not normally
taught and often are used to predict future
performance
typically provide an overall score, a verbal
score, and a quantitative score
7
2. Affective instruments...

• Measure characteristics of individuals along a
  number of dimensions and to assess feelings,
  values, and attitudes toward self, others, and a
  variety of other activities, institutions, and
  situations

8
Types of affective instruments...

• attitude scales

self-reports of an individuals beliefs,
perceptions, or feelings about self, others, and
a variety of activities, institutions, and
situations
frequently use Likert, semantic differential,
Thurstone , or Guttman scales
9

• values tests

measure the relative strength of an individuals
valuing of theoretical, economic, aesthetic,
social, political, and religious values
10

• personality inventories

an individuals self-report measuring how
behaviors characteristic of defined personality
traits describe that individual
11
3. Projective instruments...

• Measure a respondents feelings or thoughts to an
  ambiguous stimulus

12
Primary type of projective test...

• associational tests

participants react to a stimulus such as a
picture, inkblot or word onto which they project
a description
13
Selecting an instrument...
1. determine precisely the type of instrument
needed
2. identify and locate appropriate instruments
3. compare and analyze instruments
4. select best instrument
14
Instrument sources
Burros Mental Measurements Yearbook
Tests in Print
PRO-ED Publications Test Critiques Compendium
ETS Test Collection Database
ERIC/AE Test Review Locator
ERIC/Burros Test Publisher Directory
15
Rules governing the selection instruments...
1. the highest validity
2. the highest reliability
3. the greatest ease of administration, scoring,
and interpretation
4. test takers lack of familiarity with
instrument
5. avoids potentially controversial matters
16
Administering the instrument...
1. make arrangements in advance
2. ensure ideal testing environment
3. be prepared for all probable contingencies
17
Two issues in using instruments...
1. Validity the degree to which the instrument
measures what it purports to measure
2. Reliability the degree to which the
instrument consistently measures what it purports
to measure
18
Types of validity...
1. Content validity
2. Criterion-related validity
3. Construct validity
19
1. Content validity the degree to which an
instrument measures an intended content area
20
forms of content validity
sampling validity does the instrument reflect
the total content area?
item validity are the items included on the
instrument relevant to the measurement of the
intended content area?
21
2. Criterion-related validity an individual
takes two forms of an instrument which are then
correlated to discriminate between those
individuals who possess a certain characteristic
from those who do not
22
forms of criterion-related validity
concurrent validity the degree to which scores
on one test correlate to scores on another test
when both tests are administered in the same time
frame
predictive validity the degree to which a test
can predict how well individual will do in a
future situation
23
3. Construct validity a series of studies
validate that the instrument really measures what
it purports to measure
24
Types of reliability...
1. Stability
2. Equivalence
3. Internal consistency
25
1. Stability (test-retest) the degree to which
two scores on the same instrument are consistent
over time
26
2. Equivalence (equivalent forms) the degree
to which identical instruments (except for the
actual items included) yield identical scores
27
3. Internal consistency (split-half reliability
with Spearman-Brown correction formula ,
Kuder-Richardson and Cronbacks Alpha
reliabilities, scorer/rater reliability) the
degree to which one instrument yields consistent
results
28
Terms associated with instruments...
29
Data
the pieces of information researchers collect
through instruments to examine a topic or
hypothesis
30
Constructs
abstractions of behavioral factors that cannot
be observed directly and which researchers invent
to explain behavior
31
Variable
a construct that can take on two or more values
or scores
32
Raw scores
the number of items an individual scored on an
instrument
33
Measurement scales
the representation of variables so that they can
be quantified
34
Measurement scales...
Qualitative (categorical)
1. nominal variables
Quantitative (continuous)
2. ordinal variables
3. interval variables
4. ratio variables
35
1. nominal (categorical) classifies persons or
objects into two or more categories
36
2. ordinal (order) classifies persons or
objects and ranks them in terms of the degree to
which those persons or objects possess a
characteristic of interest
37
3. interval ranks, orders, and classifies
persons or objects according to equal differences
with no true zero point
38
4. ratio ranks, orders, classifies persons or
objects according to equal differences with a
true zero point
39
Norm reference
provides an indication about how one individual
performed on an instrument compared to the other
students performing on the same instrument
40
Criterion reference
involves a comparison against predetermined
levels of performance
41
Self reference
involves measuring how an individuals
performance changes over time
42
Operationalize
the process of defining behavioral processes
that can be observed
43
Standard error of measurement
an estimate of how often a researcher can expect
errors of a given size on an instrument
44
Mini-Quiz

• True or false

a large standard error of measurement indicates
a high degree of reliability
false
45

• True or false

a large standard error of measurement indicates
low reliability
true
46

• True or false

most affective tests are projective
false
47

• True or false

the primary source of test information for
educational researchers is the Burros Mental
Measurements Yearbook
true
48

• True or false

research hypotheses are usually stated in terms
of variables
true
49

• True or false

similar to a Thurstone scale, a Guttman scale
attempts to determine whether an attitude is
unidimensional
true
50

• True or false

validity requires the collection of evidence to
support the desired interpretation
true
51

• True or false

researchers should first consider developing an
instrument rather than utilizing a published
instrument
false
52

• True or false

a researchers goal is to achieve perfect
predictive validity
false
53

• True or false

predictive validity is extremely important for
instruments that are used to classify or select
individuals
true
54

• True or false

a high validity coefficient is closer to 1.00
than 0.00
true
55

• True or false

norm reference and criterion reference are
synonymous terms
false
56

• True or false

criterion related refers to correlating one
instrument with a second instrument the second
instrument is the criterion against with the
validity of the second instrument is judged
false
57

• True or false

a valid test is always reliable but a reliable
test is not always valid
true
58

• True or false

it is difficult to state appropriate reliability
coefficients because reliability, like validity,
is dependent upon the group being tested, i.e.,
groups with different characteristics will
produce different reliabilities
true
59

• True or false

content validity is not compromised if the
instrument covers topics not taught
false
60

• Fill in the blank

the tendency of an individual to respond
continually in a particular way
response set
61

• Fill in the blank

a study which consists of two quantitative
variables
correlational
62

• Fill in the blank

a study which consists of one categorical and
one quantitative variable
experimental or causal-comparative
63

• Fill in the blank

a study which consists of two or more
categorical variables
correlational or descriptive
64

• Fill in the blank

data collection methods which emphasize student
processes or products
performance
65

• Fill in the blank

data collection methods including
multiple-choice, true-false, and matching
selection
66

• Fill in the blank

data collection methods in which students fill
in the blank, provide a short answer, or write an
essay
supply
67

• Fill in the blank

an instrument administered, scored, and
interpreted in the same way no matter where or
when it is administered
standardized
68

• Fill in the blank

the term that includes the general process of
collecting, synthesizing, and interpreting
information, whether formal or informal
assessment
69

• Fill in the blank

a formal, systematic, usually paper-and-pencil
procedure for gathering information about
peoples cognitive and affective characteristics
test
70

• Fill in the blank

the degree to which individuals seek out or
participate in particular activities, objects,
and ideas
interests
71

• Fill in the blank

also called temperament, the characteristics
representing an individuals typical behaviors
and describes what individual do in their natural
life circumstances
personality
72

• Fill in the blank

things individuals feel favorable or unfavorable
about the tendency to accept or reject groups,
ideas, or objects
attitudes
73

• Fill in the blank

deeply held beliefs about ideas, persons, or
objects
values
74

• Fill in the blank

requires administering the predictor instruments
to a different sample from the same population
and developing a new equation
cross-validation
75

• Which type of test

Minnesota Multiphasic Personality Inventory
personality inventory
76

• Which type of test

Stanford-Binet
achievement test
77

• Which type of test

Strong Campbell
interest inventory
78

• Which type of test

SRA Survey of Basic Skills
achievement test
79

• Which type of test

Weschler Intelligence Scales
aptitude test
80

• Which type of test

Gates-McGinitie Reading Test
achievement test
81

• Which type of test

Otis-Lennon School Ability Test
aptitude test
82

• Which type of test

Kuder Occupational
interest inventory
83

• Which type of test

Rorschach Inkblot Test
projective
84

• Which type of test

Meyers-Briggs Type Indicator
personality inventory
85

• Which type of test

Iowa Test of Basic Skills
achievement test
86

• Which type of test

Thematic Apperception Test
projective
87

• Which type of validity

compares the content of the test to the domain
being measured
content
88

• Which type of validity

Graduate Record Examination
predictive
89

• Which type of validity

correlates scores from one instrument to scores
on a criterion measure, either at the same or
different time
criterion-related
90

• Which type of validity

amasses convergent, divergent, and
content-related evidence to determine that the
presumed construct is what is being measured
construct
91

• Which type of reliability

scores on one instrument are consistent over time
stability (test-retest)
92

• Which type of reliability

the extent to which independent scorers or a
single scorer over time agree on the scoring of
an open-ended instrument
scorer/rater
93

• Which type of reliability

scores correlate between similar version of an
instrument given at different times
equivalence and stability
94

• Which type of reliability

scores correlate between two versions of a test
that are intended to be equivalent
equivalence (alternate forms)
95

• Which type of reliability

the extent to which items included on an
instrument are similar to one another in content
internal consistency
96

• Which type of response scale

an individual gives a quantitative rating to a
topic where each position on the continuum has an
associated score value
semantic differential
97

• Which type of response scale

value points are assigned to a participants
responses to a series of statements
Likert
98

• Which type of response scale

participants select from a list of statements
that represent differing points of view from
those which participations agree
Thurstone
99
This module has focused on...
instruments
which describes the procedures researchers use
to select individuals to participate in a study
100
The next module will focus on...
qualitative research
...the tools researchers use to gather data for a
study