Frame

1
Frame
Critical notions in Measurement
2
Meet little Lidona, 4th grader
3
State Reading

• Last year, Lidona did not meet the state
  benchmark in reading
• Domain scores were low across the board
• Note making inferences based on these domain
  results is not justified (i.e., not valid).
• Candy Question Why?

4
Classroom Report

• Lidona meets most classroom expectations

What next?
5
Reading Curriculum Based Measures

• 2-3 times a year, Lidona also receives a reading
  fluency, vocabulary, and reading comprehension
  test from the district

6
Reading Curriculum Based Measures
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
Data Driven Decision Making

• We looked at the state score, the classroom
  report, and the local measure
• Little Lidona passed the state test, but is at
  the bottom of her class, and near the 25th
  percentile in the district.
• Using multiple sources of data can lead to better
  decision making

11
My First Decision

• Who are you what do you need?

12
My Data Driven Plan
Who are you what do you need?
Test you and solicit feedback
Online survey
Plots
Modify the agenda
13
Gather
(15 minutes to create)
14
Gather
(1 minute to download data)
15
Analysis
(15 minutes to generate plots) (Data displays are
part of my analysis toolbox)
16
Who responded?
17
How old were respondents?
18
given choice, most preferred color?
19
Who are you?
20
Experience
21
Does this make sense?

• The most common person is,
• Female
• Under 40
• Likes blue
• Has two kids

22
Computers
23
Excel by Gender
24
Statistics
25
Statistics by Gender
This difference was statistically different (p lt
.05)
26
p-values
27
Standard Deviation
28
t-test
29
ANOVA
30
SPSS
31
Regression
32
Research Design
33
Reliability
34
Validity
35
Sampling
36
Using Graphs
37
Making Graphs
38
Frequency of Use
39
Differences

• In general there were no significant differences
  on any question based on gender, age, or favorite
  color
• EXCEPT
• Women were less confident about their general
  statistics knowledge (p lt .05) why?
• People with fewer kids reported teaching graphs
  to others more often (p lt .05) why?

40
What did I learn?

• You dont need guidance with
• Computers
• Excel
• Using graphs
• You might need guidance with
• Statistics
• Research design
• Developing Graphs

All of these are skills necessary in measurement
and assessment
41
Review
Who are you what do you need?
Test you and solicit feedback
Online survey
Plots
Modify the agenda
42
Big Point 1

• A valuable framework for making decisions with
  data is the 5-step cycle that includes,
• Identify
• Gather
• Analyze
• Implement
• Evaluate

43
Candy Question

• What are the five stages of the D3M cycle?

44
Questions in the Cycle

• Embedded within this cycle is an assumption that
  questions identified are meaningful
• Good questions can be framed within 6 dimensions
  of student data
• Good questions can be structured so that they are
  measurable

45
6 Dimension of Student Data
Are the kids learning? Are the kids reading well?
Do the kids understand content?
achievement
How many expulsions? How many suspensions? How
many referrals? Dropouts?
behavior
How old are the kids? What income level? What
gender? What language do they speak?
demographics
time
organization
How long is lunch? How is recess scheduled? How
are classes grouped? How is allocated?
conation/climate
Are kids motivated? Are they experienced with
computers? Do kids feel safe? Do kids have a
supportive community?
46
Example Use of CBM
Do small districts use CBMs?
achievement
demographic
A survey of 17 districts in Oregon
47
Does the change in the 4th grade parents view of
the reading curriculum since last year depend on
their childs reading score?
48
Good Questions are Measurable

• Does spending an additional 20 minutes a day
  reading silently in the classroom improve 9th
  grade student scores on statewide reading tests?
• Do 2nd grade students who eat chocolate 1 hour
  before reading class improve scores on ORF tests
  faster than students who dont eat chocolate?
• Do students prefer using electronic readers
  (eBooks) to print texts with identical content?
• Do students retain more of the information
  presented in class lectures when they are
  required to reflect on that information in
  writing at least once per hour during lectures?

49
Poor Questions are Open Ended

• What can Luke do to make students pay attention
  in class?
• How can Luke deliver this material to make it
  more interesting?
• Is there any way Luke can get more money for his
  work?

50
Making a Good Question
IV
outcome
DV
sample conditions



Is the program
effective at increasing
reading scores
for 3rd grade students?
Is the intervention
effective at reducing
poor behaviors
at elementary schools?
Does training
facilitate
data comprehension
for administrators in OR?




Imagine you are a principal. Take 5 minutes to
write down a question in this format. Make it one
that will be useful in your job. Share it with a
neighbor, and check for clarity. Write one of the
questions on the board.
51
Big Point 2

• A complex multidimensional and measurable
  approach to questioning drives good decision
  making.

52
Candy Question

• Name three of the six data dimensions.

53
3 Perspectives (Referents)

• In measurement the question we often need to ask
  is, to what will we compare our data?
• There are three corresponding referents,
• This organization is not always perfect, but
  reminds us to address a referent in questioning
  and analysis

54
Norm Referenced
55
Criterion Referenced
56
Criterion Referenced
57
Criterion Referenced
skill - addition
skill - subtraction
more.dig.carry
more.dig.carry
one.dig.carry
two.dig.carry
one.dig.carry
two.dig.carry
one.dig
one.dig
two.dig
two.dig
Z 0
Z 0
58
Individually Referenced
5th grade individual student report - ORF
59
Not Always a Perfect Fit
Does the change in the 4th grade parents view of
the reading curriculum since last year depend on
their childs reading score? compared to what?
60
Big Point 3

• Although not always a perfect fit, comparing
  results to norm, criterion, and individual
  referents gives our decisions perspective.

61
Candy Question

• Give an example of a criterion referent.

In a group of 4, use the provided example to
create a norm, criterion, and individual
referent. Write the referents on the board.
62
Three Big Points
Decision Making
Evaluation System
achievement
Norm
Identify
behavior
Criterion
Gather
demographics
time
Individual
Analyze
organization
Implement
conation/climate
Evaluate
IV/outcome/DV/sample
perspective
dimension
logic frame