High School Teachers Instructional Use of WASL Data: Exploring the Role of School Culture and Motiva

1
High School Teachers Instructional Use of WASL
Data Exploring the Role of School Culture and
Motivation

• Jack B. Monpas-Huber, Ph.D.
• Director of Assessment and Program Evaluation
• Spokane Public Schools

2
Who I Am
Master of Science, Sociology, 1997
Ph.D., Educational Psychology
Director of Assessment and Program Evaluation
3
Acknowledgments
4
Background of the Project

• Work experience
• Assessment department of large school district
• Providing data to schools to support DBDM
• Dealing with school cultures, politics,
  leadership
• Research interests
• Sociology of education / school organization
• Motivation
• Measurement, statistics research design
• Validating large-scale accountability systems

5
Organization of this Presentation

• Framing the Problem
• Teachers Use of State Assessment Data
• Research Methods
• Results of the Study
• Discussion

6
Framing the Problem

• Rise of state accountability programs
• High stakes attached to student performance
• Data fed back to schools for data-based
  decisionmaking
• Theory of Action research (Fuhrman, 2004)
• Two functions
• Accountability
• Instructional/feedback
• How do the two forces shape teachers
  instructional use of data?

7
Research Questions

• Considering how much data the state provides to
  educators, how much are high school teachers
  using state assessment data as a resource to
  improve instruction? How useful do they find it?
• Considering the mounting policy pressures to
  improve performance on the state assessment, what
  motivates teachers to use state assessment data?
  What is the influence of policy pressure
  specifically, and aspects of school context in
  general?

8
Limits in Focus

• State Assessment data
• Instructional decisions
• Certificated teachers
• High schools

9
Organization of this Presentation

• Framing the Problem
• Teachers Use of State Assessment Data
• Research Methods
• Results of the Study
• Discussion

10
Teachers Use of State Assessment DataRelevant
Literatures

• Accountability Systems
• Data-based Decision-making
• Teacher Motivation
• Accountability and High Schools

11
Teachers Use of State Assessment DataCapacity
for Teacher Data Use

• Technical Skills
• Technical skills for working with data
• Databases, software
• Analysis and interpretation of systematically
  collected data
• Capacity building efforts in Washington
• Hypothesis
• Exposure to training in assessment or in WASL
  item development should be a strong predictor of
  use of WASL data

12
Teachers Use of State Assessment DataCapacity
for Teacher Data Use

• Access to Data
• Advances in computer technology
• Assessment personnel
• Teachers may vary in their perception of access
  to data
• Hypothesis
• Teachers who perceive more access to data should
  be more likely to use such data than teachers who
  perceive less access to data
• Access as necessary but not sufficient condition

13
Teachers Use of State Assessment DataTeacher
Motivation and Data Use

• The Policy Perspective
• Some instructional changes are difficult
• Teachers need consequences
• Behavioral perspective on motivation
• Research on high stakes testing
• Research issue perceived pressure as both
  outcome and predictor
• Hypothesis
• Teachers who perceive higher levels of pressure
  will be more likely to use assessment data than
  teachers who perceive lower levels of pressure

14
Teachers Use of State Assessment DataTeacher
Motivation and Data Use

• Alternative Perspectives
• Cognitive perspectives on motivation
• Motivation stems from mind/thought/interpretation
• Social context and cognition

15
Teachers Use of State Assessment DataTeacher
Motivation and Data Use

• Expectancy
• Teachers perceived probability that the
  teachers effort will result in the attainment of
  the goals (Kelley, Heneman, Milanowski, 2002,
  p. 378)
• Will do of motivation efforts will result in
  positive outcomes
• Kentucky and North Carolina research
• Hypothesis
• Teachers who report higher levels of expectancy
  (that working with assessment data will actually
  help them improve instruction for their students)
  will be more likely to use assessment data than
  those who expect less to result from it

16
Teachers Use of State Assessment DataTeacher
Motivation and Data Use

• Efficacy
• Teacher beliefs about their span of influence
  and performance capacity (Kelley Finnigan,
  2003, p. 604)
• Can do of motivation
• The role of performance feedback in motivation
  research
• Hypothesis
• Teachers who feel more efficacious working with
  assessment data will be more likely to use
  assessment data than teachers who feel less
  efficacious

17
Teachers Use of State Assessment DataTeacher
Motivation and Data Use

• Goals
• Motivation as product of intentions or goals
  people have for engaging in a behavior
• People pursue variety of goals
• Goals may conflict with each other
• Teachers and perceived policy intentions of
  accountability policies (Leithwood, Steinbach,
  Jantzi, 2002)
• Ingram, Louis, Schroeder (2004) study
• Hypothesis
• Teachers will be more likely to use state
  assessment data if they perceive its underlying
  purpose as consistent with their own goal of
  helping students learn

18
Teachers Use of State Assessment DataSummary of
Motivation Research

• Pressure, expectancy, efficacy, goals
• Filtered through school context
• Some aspects of context (collaboration, feedback
  data) influence motivations
• These motivations vary
• Among teachers within one school
• Possibly by groups of teachers between schools
• Research issues
• Motivations as predictors of data use
• Motivational effects may be different in
  different schools

19
Teachers Use of State Assessment DataBuilding a
Model of Teacher Data Use

• Quantity of Teacher Data Use
• ß0 (mean)
• ß1(state assessment training)
• ß2(perceived access to data)
• ß3(perceived pressure)
• ß4(expectancy)
• ß5(efficacy)
• ß6(goal alignment)
• e (unmodeled variation)

20
Teachers Use of State Assessment DataContextual
Influences on Teacher Motivation and Data Use

• To the extent that motivations are shared by
  teachers in one school, what influences these
  motivations?
• Are some schools more motivating than others,
  in this case, in regard to using and learning
  from state assessment data?
• Sociological perspectives on school culture and
  other contextual influence

21
Teachers Use of State Assessment DataContextual
Influences Culture

• Focus on shared attitudes and behavior is focus
  on culture
• Two perspectives on culture (Swidler, 1995)
• Inside out internalized attitudes
  (motivations?) predict behavior
• Outside in shared practice, norms, codes
  regulate behavior irrespective of internal beliefs

22
Teachers Use of State Assessment DataCultural
Perspectives

• The Loose Coupling Perspective (Weick, 1976
  Firestone, 1985)
• Educational organizations are multi-layered
• Classrooms disconnected from administration
• Because teaching and learning is not precise,
  schools do not evaluate technical quality of
  instruction
• High schools especially loosely coupled
• Challenges bureaucratic models of schools which
  emphasize centrality of leadership and formal
  rational procedures
• Also helps explain why reform movements have
  historically failed to change instruction in
  schools
• Loose coupling and assessment data
• Stick them in a drawer

23
Teachers Use of State Assessment DataCultural
Perspectives

• Professional Accountability
• Abelmann Elmore (1999)
• Strong and weak internal accountability systems
• ODay (2004)
• Professional Collaboration
• Student learning data as centerpiece of
  collaborative work
• Recurrent predictor in past research

24
Teachers Use of State Assessment DataLeadership

• Leaders filter and frame accountability policy
  (Spillane)
• Transformational leadership has positive effects
  on teacher motivation
• Trust, collaboration, shared accountability
• Principals may vary in how they frame assessment
  results
• School-level variable that influences motivations
  and assessment data

25
Teachers Use of State Assessment
DataMethodological Observations

• Lots of qualitative case studies
• No quantitative studies of use as a criterion or
  dependent variable
• Lack basic descriptive data about levels or
  frequencies of use

26
Teachers Use of State Assessment DataA
Tentative Model
27
Organization of this Presentation

• Framing the Problem
• Teachers Use of State Assessment Data
• Research Methods
• Results of the Study
• Discussion

28
Research MethodsDesign Issues

• Teacher survey
• Study population certificated teachers in high
  schools in western Washington school districts
  that employ a full-time assessment director
• Instrument 4-page questionnaire
• Matrix sampling
• Three forms
• Each contained common and unique items

29
Research MethodsSchool Sample Characteristics
Final sample size for analysis 376 WASL
teachers (teach 10th grade AND math, English,
science, special education, or ELL)
30
Research MethodsSample Characteristics Teachers
31
Research MethodsScale Development

• Classical Test Theory
• Internal consistency reliability (Cronbachs
  coefficient alpha (a))
• Item-total correlations
• Exploratory factor analyses (EFA)
• Item Response Theory
• Rating Scale Model (Wright Masters, 1982)
• Item difficulty, fit statistics

32
Research MethodsOutcome Measures

• Frequency of WASL Data Use
• Utility of WASL Data Use

33
Research MethodsPredictor Measures

• Perceived Access to Data
• Training in State Assessment
• Training in WASL Item Construction
• Pressure to Increase WASL Performance
• WASL Goal Alignment
• Efficacy with WASL Data
• Principal WASL Commitment
• Principal Trust
• Departmental Professional Collaboration
• Professional Accountability

34
Organization of this Presentation

• Framing the Problem
• Teachers Use of State Assessment Data
• Research Methods
• Results of the Study
• Discussion

35
ResultsHow much are teachers using data?
36
ResultsHow much are teachers using data?
37
ResultsHow much are teachers using data?
38
ResultsHow much are teachers using data?
39
ResultsHow much are teachers using data?
40
ResultsHow much are teachers using data?
41
ResultsHow much are teachers using WASL data?
Range 0-100 Mean 37.5 SD 13.9
What does this scale mean in practical terms?
42
ResultsHow much do teachers benefit from WASL
data?
43
ResultsHow much do teachers benefit from WASL
data?
44
ResultsHow much do teachers benefit from WASL
data?
45
ResultsHow much do teachers benefit from WASL
data?
46
ResultsHow much do teachers benefit from WASL
data?
WASL data are of use to me in my instruction
47
ResultsHow much do teachers benefit from WASL
data?
48
ResultsHow much do teachers benefit from WASL
data?
49
ResultsHow much do teachers benefit from WASL
data?
Range 0-100 Mean 46.4 SD 15.1
50
ResultsWhat motivates teachers to use WASL data?
51
ResultsVisualizing Hierarchical Linear Modeling,
1/3
r 0.52
52
ResultsVisualizing Hierarchical Linear Modeling,
2/3
53
ResultsVisualizing Hierarchical Linear Modeling,
3/3
Frequency of WASL Data Use (0-100)
Pressure to Increase WASL Performance (0-100)
54
ResultsFrequency of WASL Data Use HLM Results
Intraclass correlation coefficient uoj / uoj
rij .11 11
55
ResultsFinal Model of Frequency of Data Use
HLM Results

• Frequency of WASL Data Use
• ß0 (mean)
• ß1(utility of WASL data use)
• ß2(training in WASL item writing)
• ß3(perceived pressure to increase WASL scores)
• ß4(principal commitment to WASL improvement)
• ß5(efficacy with WASL data)
• r (unmodeled variation)
• ß0 ?00 ?01(school ethnic composition) u0

56
ResultsModeling Frequency of WASL Data Use
57
ResultsUtility of WASL Data Use HLM Results
Intraclass correlation coefficient uoj / uoj
rij .008 0.8
58
ResultsFinal Model of Utility of Data Use HLM
Results

• Utility of WASL Data Use
• ß0 (mean)
• ß1(perceived access to WASL data)
• ß2(frequency of WASL data use)
• ß3(departmental professional accountability)
• ß4(departmenal professional collaboration)
• ß5(WASL goal alignment)
• ß6(WASL efficacy)
• r (unmodeled variation)

59
Organization of this Presentation

• Framing the Problem
• Teachers Use of State Assessment Data
• Research Methods
• Results of the Study
• Discussion

60
Conclusions

• High School Teachers Use of Data
• Teachers are using data with moderate frequency
  and gaining some value from it
• This aspect of WASL program is working
• Motivation and School Context
• Multiple motivations are at work (pressure and
  efficacy)
• Principal leadership provides incentive to use
• Sensemaking of data is social / collaborative
• Data as feature of more tightly coupled schools

61
Thank you!

• To contact me
• Jack B. Monpas-Huber, PhD
• Director of Assessment and Program Evaluation
• Spokane Public Schools
• jackm_at_spokaneschools.org
• [phone] Office
• [phone] Cell