Reliability and Sampling

1
Reliability and Sampling

• Dennison S. Bhola, Ph.D.

2
What is this a picture of?
3
What is this a picture of?
4
What is this a picture of?
5
What is this a picture of?
6
Do you need more information?

• Would you be in a better position to identify the
  object if you had more information?
• Lets see

7
What is this a picture of?
8
What is this a picture of?
9
What is this a picture of?
10
Observation

• As the amount of information we have increases,
  the reliability of our judgments increases.

11
Observation

• By improving reliability and focusing on
  important content we can make more accurate
  (valid) decisions.
• Thus improving reliability can result in
  improving validity.

12
Lets try another
13
Guess again Whats this?
14
Try again What is this a picture of?
15
What is this a picture of?
16
What is this a picture of?
17
What is this a picture of?
18
What is this a picture of?
19
What is this a picture of?
20
Lets try another
The slide contains 3 objects
21
What is this a picture of?
22
What is this a picture of?
23
What is this a picture of?
24
What is this a picture of?
25
What is this a picture of?
26
What is this a picture of?
27
What is this a picture of?
28
What is this a picture of?
29
What is this a picture of?
30
What is this a picture of?
31
What is this a picture of?
32
What is this a picture of?
33
Apple, red ball, tomato
34
Now let us make a connection between reliability
and sampling

• Each assessment we create contains questions.
• These questions are a sample of the possible qs
  we can ask in a particular content domain.

35
Reliability and sampling

• With qs of similar quality
• the more qs we ask ? the more information we
  gather, and the higher the reliability of the
  assessment.

36
Samples of questions sampling and content
validity

• Samples that are representative of a population
  allow us to use statistics to make
  generalizations to that population.
• Keep this in mind when choosing the qs for your
  assessment. A Table of Specifications is useful
  here (more on this later).

37
Samples of people

• In research we use samples of people from a
  target population, because it is often too costly
  to include the whole population in the study.
• Samples that are representative of a population,
  allow us to use statistics to make inferences
  about that population.

38
Samples

• Samples must be carefully selected to be
  representative of the population we want to
  generalize our research findings to.
• Consider the following scenarios

39
If your sample is made up of 5 volley balls, what
population can you make generalizations to?
40
The 5 volley balls are a good sample for this
population. You could make generalizations to
the volley ball population based on measurements
of the sample.
41
The 5 volley balls are NOT a good sample for this
population. You cannot make generalizations to
this population based on measurements of the
volley ball sample.
42
To make generalizations to the previous
population, this would be a better sample to use.
43
In order to use measurements on a sample to make
generalizations to a population, the sample must
be representative of the population.
44
Reliability of Tests

• Depends upon
• The number of good quality test items used
• more good items provide more information and the
  reliability is likely to improve.

45
Content Validity

• When you develop a test
• By design, the test items should sample the
  content domain that you want to make inferences
  about.

46
Inferences

• In order to make inferences about the content
  domain of interest, the set of items on your test
  must be representative of that content domain.

47
Summary

• Reliability increases with the number of high
  quality items on an assessment.
• High quality items provide valuable information
  about student learning.

48
Summary

• Appropriate sampling of a content domain enables
  us to make inferences about student learning in
  that content domainthis is the essence of
  Content Validity.

49
Summary

• High reliability combined with appropriate
  sampling of the target content domain are
  important for assessment results to lead to valid
  decisions.