Development and Use of a Tool for Evaluating Teacher Effectiveness

1
Development and Use of a Tool for Evaluating
Teacher Effectiveness

• Alysia D. Roehrig
• Educational Psychology Learning Systems
• Florida Center for Reading Research
• Florida State University

2
Exemplary Teaching Studies

• Inductive grounded theory approach used to gain
  unique insights into teaching what exemplary
  primary teachers do

• Pressley, Rankin Yokoi, 1996
• Pressley, Wharton-McDonald,
• Allington, Block, Morrow, Tracey,
• Baker, Brooks, Cronin, Nelson
• Woo, 2001
• Roehrig, Pressley Sloup, 2001
• Wharton-McDonald, Pressley
• Hampston, 1998

• Bogner, Raphael Pressley, 2002
• Day, Boothroyd, Johnston Cedeno, 1999
• Day, Woodside-Jiron Johnston, 1999
• Dolezal, Welsh, Pressley Vincent, 2003
• Johnston, Powers Costello, 1999
• Morrow, Tracey, Woo Pressley, 1999

What sets exemplary teachers apart?
3
Three Levels of Effectiveness

• Effective Teachers Students
• Consistently engaged in reading and writing
• Higher achievement in reading and writing could
  write many pages with good mechanics and spelling
  and read books above grade level by end of year
• Typical Teachers Students
• Variable student engagement
• Reading and writing achievement less impressive
• Ineffective Teachers Students
• Consistently low engagement
• Reading and writing achievement even less
  impressive

4
Exemplary Teachers had

• Classrooms consistently saturated with dozens of
  motivational practices no undermining ones
• (Bogner et al., 2002 Dolezal et al., 2003)

5

Developing the Classroom AIMS Instrument

• Qualitative teacher studies results inductively
  coded to develop categories of items for
  instrument
• Descriptive headings
• Patterns/categories
• Reexamine original data confirm or recategorize
• Look for connections between categories

6
Developing Validating the AIMS Instrument

• 3 rounds of coding completed to confirm
  categorization
• Surveyed expert researchers teachers about
  importance of each practice/item to exemplary
  teaching
• Initially used instrument to assess teaching
  practices of 20 teachers

7
Rating with the AIMS Instrument

• Recommend 2 people observe class
• Extensive field notes taken of all interactions
• AIMS items rated independently
• 1seldom, 2somewhat, or 3consistently
  characteristic of those found in exemplary
  classrooms
• Disagreements resolved

8
Content of the AIMS Instrument

• 4 major categories 13 subcategories
• Atmosphere
• Instruction/Content
• Management
• Student Engagement

9
Atmosphere

• Everything the teacher does that affects the
  physical interpersonal environment to get and
  keep students involved in learning.

10
Atmosphere

• Sense of Community Fostered
• Teacher expresses empathy for students-encouraging
  others to do so as well.
• There is a high use of personal pronouns
  communicating a sense of community,
  participation, and ownership (e.g., This is OUR
  class. WE will work together.).

11
Atmosphere

• Interest Fostered
• Teacher encourages curiosity/suspensegetting
  students excited about what they are
  learning/doing (e.g., Listen carefully to the
  story, youll find out the answer to our
  questions. Tomorrow we are going to be having
  a special mystery visitor!).

12
Atmosphere

• Focus on Effort Rather than Performance
• Teacher attributes success to effort (e.g.,
  strategies) and time. Given these, success was
  believed to be imminent for ALL students.
• Teacher does NOT attribute student successes and
  failures to luck, ability, or task difficulty
  (i.e., negative attributions that could
  discourage students).

13
Atmosphere

• Value of Learning Expressed
• Teacher encourages students to set goals for
  their reading (e.g., to read so many books this
  month, typically, a few more than were read last
  month).

14
Atmosphere

• High Expectations Expressed
• Teacher communicates to students that she is
  determined students WILL learn.
• Teacher signals difficult tasks that students CAN
  do.

15
Atmosphere

• Informative Feedback Provided
• Teacher often provides feedback that is
  informativeteacher does NOT give much
  unspecific, blanket praise.

16
Instruction/Content

• The lessons, activities, and the teachers
  instructional style.

17
Instruction/Content

• Engaging Content and Activities Used
• Students lived experience is integrated with
  literate practice (e.g., students encouraged to
  connect reading and writing with personal
  experiences).
• Tasks matched to student interests.

18
Instruction/Content

• High Instructional Density Achieved
• Teacher seems to teach constantly, with whole
  group, small group, and individual mini-lessons
  simply intermingled throughout the day.
• The teacher does NOT miss many opportunities to
  explain answers.

19
Instruction/Content

• Appropriate Challenge Level Achieved
• Teacher supports appropriate risk-taking (i.e.
  students are encouraged to take on
  instructionally challenging tasks instead of ones
  that are too easy or too hard for them).

20
Instruction/Content

• Thinking Processes Modeled and Taught
• Teacher explicitly articulates the processes used
  in strategies/problem solving (e.g., explains the
  steps involved in brainstorming).

21
Instruction/Content

• Academic Monitoring Provided
• Teacher walks around the room and does lots of
  monitoring of the students as they are working in
  order to check for understanding.

22
Management

• The order, rules, routines, procedures what
  keeps the instruction moving in an orderly
  fashion.

23
Management

• Behavioral Self Regulation Encouraged
• Teacher effectively uses redirection (e.g. asking
  students what they are doing or what they should
  be doing), glances, pauses to help keep students
  on-task.

24
Management

• Behavior/Task Monitoring Provided
• Teacher does whole class monitoring for on-task
  behavior.
• Teacher does NOT use punishment that distracts
  students from their work.

25
Student Engagement

• Students vocalize/express excitement about
  content/activities (e.g., lots of Oohs and
  Aahs.).
• The students eagerly raise their hands and
  participate.
• At least 80 of students are consistently on task
  and highly engaged in class activities.
• Students are so self-regulated that disciplinary
  encounters are rarely seen.

26
AIMS in Action

• Can and has been used a number of different ways
• More on that later
• Primarily to evaluate teacher effectiveness/change

27
Data Sources

• N 109
• Primarily elementary grades
• Between 1-47 years teaching experience
• Multiple states and countries
• 22 Public and Private schools

28
Internal Consistency Reliability

• All subcategories of teaching practices a gt .63
• Atmosphere category a .87
• Instruction/Content category a .90
• Management category a .74
• Student Engagement category a .79
• Range 1-3, M 2.23, SD .53

29
Distribution of Atmosphere Scores
30
Distribution of Management Scores







31
Distribution of Instruction/Content Scores
32
Correlations with Student Engagement (p lt .01)

• Atmosphere r 0.64
• Instruction/Content r 0.67
• Management r 0.76

33
Using the AIMS Instrument to Improve Teaching

• Theoretical organization of exemplary teaching
  practices
• To evaluate teaching effectiveness/change
• To organize professional development
• To code interviews, other data
• To guide reflection observation

34
Evaluating Teachers

• Bohn, Roehrig, Pressley (2004). The first days
  of school in effective and less effective
  primary-grades classrooms. Elementary School
  Journal, 104, 269-287.
• AIMS used to evaluate classrooms
• Qualitative study

35
Evaluating Teachers

• Effective teachers
• did more to establish routines procedures at
  beginning of year,
• provided more engaging activities,
• indicated higher expectations,
• and did more to encourage student self-regulation

36
Guiding PD

• Roehrig, Bohn, Turner, Pressley (2008).
  Mentoring beginning primary teachers for
  exemplary teaching practices. Teaching and
  Teacher Education, 24, 684-702.
• AIMS used to guide beginning teacher induction/PD
  
• AIMS also used to evaluate teachers for
  effectiveness change
• Qualitative study

37
Guiding PD

• Explored the potential for mentoring to support
  novice teachers use of effective teaching
  practices
• More effective beginning teachers mentors had
  more experience as mentors and were more
  effective teachers than other mentors
• More effective beginning teachers communicated
  more with mentors, more accurately self-reported
  use of effective teaching practices, and were
  more open to mentoring

38
To Code Interviews

• Roehrig, Turner, Grove, Schneider, Liu (in
  press). Degree of alignment between beginning
  teachers' practices and beliefs about effective
  classroom practices. The Teacher Educator.  
• AIMS used to code interviews observations
• Qualitative study

39
To Code Interviews

• Studied the relationships among teachers
  practices and beliefs
• Compared the presence/absence of alignment among
  these variables in 6 beginning teachers with the
  alignment of these variables in an experienced,
  exemplary teacher
• A testable model emerged depicting a
  metacognitive feedback loop for teachers who are
  aware of their shortcomings and place
  responsibility for students behaviors and
  learning on themselves

40
To Guide Reflection

• Roehrig, Guidry, Bodur, Guan, Guo, Pop (2008).
  Guided field observations Variables related to
  preservice teachers knowledge about effective
  primary reading instruction. Literacy Research
  and Instruction, 47(2), 76-98.
• AIMS used to guide preservice teachers
  observations of their supervising teachers
• AIMS also used to code their concept maps
• Correlational study

41
To Guide Reflection

• Studied the relations between preservice
  teachers guided field observations of primary
  literacy instruction and knowledge about
  effective beginning reading practices
• The more effective motivating practices a
  preservice teacher observed, the more likely he
  or she was to reflect these practices in a
  concept map representing effective beginning
  literacy instruction, r S .368, p .015
• Even as a preservice teacher more often observed
  and correctly identified both examples and
  non-examples of effective motivating practices,
  there was not a tendency for her or him to
  represent those practices in a concept map, r S
  .186, p .142

42
Future Directions

• More data from secondary teachers
• Factor analysis
• Connect AIMS scores to PD implementation and
  student outcomes

43
Questions?

• More details about the items?
• More details about previous current studies?
• Suggestions about additional uses?
• Others?

44
Thank You!

• Special thanks to Eric Christesen!!
• For more information, please email
  aroehrig_at_fcrr.org