Evaluation of an Arts-Based Instructional Program for Primary Grades:

1
Evaluation of an Arts-Based Instructional Program
for Primary Grades  Preliminary Findings and
Lessons Learned from WebPlay
Noelle Griffin, Ph.D.
American Educational Research AssociationAnnual
Conference New York March 24-28, 2008
2
Overview

• Section 1 WebPlay Evaluation Overview
• Section 2 Survey Design and Analysis
• Section 3 Pre-/Post- Comparison Results

3
Section 1WebPlay Evaluation Overview
4
WebPlay Program

• Internet-enhanced arts education project
• Integrates theater arts in school curriculum
• Performing arts, literacy, social studies, and
  technology

5
WebPlay Curriculum

• Two weekly lessons
• Use of internet Connections with partner
  classrooms
• WebPlay curriculum involves activities in the
  following four areas
• Performing Arts
• English Language Arts
• History/Social Studies
• Technology

6
Evaluation

• 3 year evaluation beginning 2005-06
• 3rd and 5th grade classes/large urban school
  district
• Quantitative methods/small scale
• Both summative and formative purposes

7
Evaluation Questions WebPlay Program

1. Will WebPlay improve student performance?
2. What is the impact of WebPlay on student skill
   development?
3. Do effects persist across different student/site
   cohorts?
4. Do different student/site characteristics
   interact with treatment?

8
Sampling and Methodology

• Sample (2006-07 school year)
• 18 WebPlay schools
• 12 matched comparison schools (for survey)
• Student survey (pre-/post-)
• Standardized state test data (English Language
  Arts) future analysis

9
Evaluation Roadblocks

• Selecting outcome measures
• The focus on standardized data
• Lack of resources
• Comparison participation

10
Section 2Survey Design and Analysis
11
Key Constructs in WebPlay Curriculum

1. Theatrical knowledge
2. Internet knowledge and safety
3. Engagement
4. Efficacy and academic esteem
5. Collaboration and broadening horizons

12
Sample Items by Construct
Areas Items
1. Theatrical knowledge The location or place (setting) of a play is where the play is being presented.
2. Internet knowledge and safety E-mails as well as the documents sent via e-mail (attachments) can have viruses.
3. School engagement I think the activities I do in school are boring.
4. Efficacy and academic esteem I would be able to do a good job if I had to write a story for school.
5. Collaboration and broadening horizons I can learn new things from students who live in other countries.
13
Factor Analysis

• Pre- surveys
• N 590 (424 WebPlay, 166 comparison)
• Exploratory and confirmatory factor analysis
  techniques

14
Exploratory Factor Analysis

• Based on Likert-style items
• Three higher order factors determined
• Theatrical engagement/interest
• General academic confidence/engagement
• External connections

15
Confirmatory Factor Analysis

• Initial fits indices mixed (i.e., not all indices
  reached cut-offs)
• Initial model revised (allow for cross-loadings,
  correlated residuals)
• Consistent with survey theoretical design
• Fit indices improved

16
WebPlay Student Survey Final CFA Model
17
Section 3 Pre/Post Comparison Results
18
Survey Pre-/Post- Analyses

• Comparing pre/post results for WebPlay and
  comparison students
• Based on only students with link-able pre/post
  data (i.e., smaller sample)
• Hierarchical linear modeling techniques used (due
  to nested nature of data)
• Two-level model (student and school)

19
Applicability of Hierarchical Linear Models
(HLMs) to Evaluation Studies

• HLM allows analysis of outcome variable
  relationship to predictors at multiple data
  levels
• Evaluation studies often involve
• Intact clusters
• Nested data
• Background characteristics of individuals that
  may vary appreciably across different clusters

20
Overall Results

• Two sets of outcomes analyzed
• Knowledge
• Attitude/Engagement
• Knowledge No significant group differences (p
  .41)
• Engagement/attitude WebPlay students increased
  pre-/post- relative to comparison students (p
  .02)

21
HM Results for Knowledge Outcome
Fixed Effects Coefficient SE p Value
Intercept 7.10 0.21 lt.0001
WebPlay -0.22 0.26 0.41
Pre-test 0.20 0.06 0.00
Random Effects Variance Component SE p Value
Adjusted means 0.18 0.10 0.03
Student residual 2.12 0.17 lt.0001
22
HM Results for Engagement/Attitude Outcome
Fixed Effects Coefficient SE p Value
Intercept 3.80 0.07 lt.0001
WebPlay 0.19 0.08 0.02
Pre-test 0.57 0.05 lt.0001
Random Effects Variance Component SE p Value
Adjusted mean 0.00 0.01 0.43
Student residual 0.36 0.03 lt.0001
23
Sub-scale Analyses

• Descriptive/exploratory in nature
• No variation in impact between types of Knowledge
  items (theater, internet)
• Some apparent variation in impact on
  attitude/engagement sub-items
• Theatrical engagement/interest WebPlay vs.
  comparison group differences most pronounced

24
Attitude Outcome sub-area scores by treatment
group
25
Next Steps

• Integration of state standardized test data
• Revision of Knowledge items (multiple choice)
• Collection and integration of teacher-level data

26
griffin_at_cse.ucla.edu