The Development and Validation of the Evaluation Involvement Scale for Use in Multi-site Evaluations Stacie A. ToalUniversity of Minnesota

1
The Development and Validation of the Evaluation
Involvement Scale for Use in Multi-site
EvaluationsStacie A. Toal University of
Minnesota
The Beyond Evaluation Use Research Team
The Beyond Evaluation Use Research Project
Funded by the National Science Foundation, this
three-year research project is studying the use
and influence of evaluations of four NSF-funded
programs by examining the relationship between
the extent of involvement of evaluation
stakeholders and the long-term impact of the
evaluations on project staff, on the science,
technology, engineering, and mathematics (STEM)
community, and on the evaluation community.
B. Volkov, S. Toal, F. Lawrenz, L. Greenseid,
K. Johnson, J. King
Messicks Aspect of Validity
Evidence Presented
Strength of Evidence Strong Weak
The Evaluation Involvement Scale
Theoretical Soundness 1. Amount of published research 2. Data from think-alouds 3. Interview data
Internal Consistency 1. Item variance 2. Scale reliability (alpha.94)
Inclusive of Relevant Activities Processes 1. Expert opinion 2. Scale items mentioned in interviews 3. Types of involvement (mentioned in literature or interviews) covered by scale
Measures Actual Involvement 1. Survey response distributions 2. Project interviews
Statistical Factors Match Rational Structures 1. Exploratory Factor Analysis 2. Confirmatory Factor Analysis
Differentiates between Groups that are Rationally or Theoretically Different 1. Significant difference between evaluators and non-evaluators in implementation, but not in planning.
Correlates as Expected with Other Variables 1. Significant, positive correlation with evaluation use.
Measures Involvement in Other Settings 1. Equally strong reliabilities for each program, but low level of involvement in one program.
Consequential Validity 1. Discussion of possible biases related to multicultural validity and evaluation use.

• Why Validate a Scale for Involvement?
• Improve research on participatory evaluation by
  making it easier for researchers to replicate
  studies in various settings.
• Identify factors critical to involvement so that
  evaluators can incorporate activities into
  multi-site evaluations to help participants feel
  involved.
• Justify the investment of time and resources in
  participatory approaches if there is more
  research available to demonstrate a positive
  relationship between involvement and evaluation
  use.
• Theoretical Framework Messicks Unitary Concept
  of Validity
• Validity is an integrated evaluative judgment of
  the degree to which empirical evidence and
  theoretical rationales support the adequacy and
  appropriateness of inferences and actions based
  on test scores or other modes of assessment.
  (Messick, 1989, 1995a, 1995b)

Question Stem To what extent were you involved
in
 

• 1. Discussions that focused the evaluation
• 2. Identifying evaluation planning team members
• 3. Developing the evaluation plan
• 4. Developing data collection instruments
• 5. Developing data collection processes
• 6. Collecting data
• 7. Reviewing collected data for accuracy and/or
  completeness
• 8. Analyzing data
• 9. Interpreting collected data
• 10. Writing evaluation reports
• 11. Presenting evaluation findings (e.g., to
  staff, to stakeholders, to an external audience)

Factor 1 Involvement in Planning
Factor 2 Involvement in Implementation
Primary Data Sources
PI Evaluator Online Survey Aug 2006 Jan
2007 Topic Involvement and use in four NSF
multi-site evaluations. Sample Size 372/810
Response Rate 46 Non-respondent study
indicated no significant differences in levels of
involvement between respondents and
non-respondents.
Interviews Mar 2007 Apr 2007 Topic
Follow-up to survey with two questions related
to involvement. Sample Size 12 respondents who
have various levels of involvement and
use.
Literature Review Fall 2005 June 2007
Topic Participatory evaluation research
studies. Scope 27 empirical and theoretical
articles directly related to participatory
evaluation.
Response Options 1No 2Yes, a little
3Yes, some 4Yes, extensively N/AThis
activity did not take place
Key Validity References Brualdi, A. (1999).
Traditional and modern concepts of validity
(Report No. ED435714). Washington D.C. ERIC
Clearinghouse on Assessment and
Evaluation. Kane, M. T. (1992). Quantitative
methods in psychology An argument-based approach
to validity. Psychological Bulletin, 112(3),
527-535. Kane, M. T. (2001). Current concerns in
validity theory. Journal of Educational
Measurement, 38(4), 319-342. Messick, S. (1989).
Meaning and values in test validation The
science and ethics of assessment. Educational
Researcher, 18(2), 5-11. Messick, S. (1995a).
Standards of validity and the validity of
standards in performance assessment. Educational
Measurement Issues and Practice, 14(4),
5-8. Messick, S. (1995b). Validity of
psychological assessment Validation of
inferences from persons' responses and
performances as scientific inquiry into score
meaning. American Psychologist, 50(9), 741-749.
Implications of this Research

• Validated evaluation research tool
• 11-item survey on evaluation involvement in
  multi-site evaluations
• 2 factors planning and implementation
• Improved understanding of what involvement looks
  like in multi-site evaluations
• Data collection
• Attendance at meetings
• Facilitates future research stakeholder
  involvement
• Positively correlated to evaluation use
• Needs to be validated in different evaluation
  contexts