THE OBJECTIVES-ORIENTED EVALUATION APPROACH Case Study: When Hard Questions Are Asked: Evaluating Writing Centers. James H. Bell

1
THE OBJECTIVES-ORIENTED EVALUATION APPROACHCase
Study When Hard Questions Are Asked Evaluating
Writing Centers. James H. Bell

• Gabriel M. Telleria
• Program Evaluation and Policy Analysis
• Spring 2008
• January 23, 2008

2
THE OBJECTIVES-ORIENTED EVALUATION APPROACH

• Why is evaluation important?
• Why do we evaluate?
• Evaluation is the systematic collection and
  interpretation of evidence, leading, as part of
  the process, to a judgment of value with a view
  to action (Beeby, qtd. In Wolf 3, qtd. in Bell
  8, 2000).

3
THE OBJECTIVES-ORIENTED EVALUATION APPROACH

• Establish broad goals or objectives
• Classify the goals or objectives
• Define objectives in behavioral terms
• Find situations in which achievement of
  objectives can be shown
• Develop or select measurement techniques
  (indicators)
• Collect performance data
• Compare performance data with behaviorally stated
  objectives

4
Advantages

• focus on outcomes
• ease of use, simplicity
• forces objectives to be set
• widespread acceptability

5
Disadvantages

• oversimplification
• outcomes-only orientation
• may neglect the value of the objectives
  themselves
• may overlook unintended outcomes

6
Case Study When Hard Questions Are Asked
Evaluating Writing Centers. By James H. Bell

• Should writing centers evaluate themselves on a
  regular basis?
• What is the best-type of evaluation approach for
  a Writing Center?
• What will an evaluation achieve?

7
What is the best-type of evaluation approach for
a Writing Center?

• Evaluation is the systematic collection and
  interpretation of evidence, leading, as part of
  the process, to a judgment of value with a view
  to action (from Bell, 2000).
• 4 important aspects of evaluations for writing
  centers
• Systematic
• Interpretation
• Judgment
• Action

8
What is the purpose of the evaluation?

• Improvement
• Audience program personnel
• Formative evaluation
• i.e.
• Funding
• Audience Supervisors, consumers, funders
• Summative evaluation

9
So how should writing centers be evaluated?

• Consumer-oriented,
• Adversary-oriented
• Management-oriented
• Naturalistic and participant-oriented
• Expertise oriented
• Objectives oriented

10
Objectives-Oriented

• Writing centers are essentially trying to
  accomplish things.
• Writing centers are aiming to alter behavior
• Objectives-oriented evaluation specializes in
  documenting behavior change.
• Speaks to administrators and funders.

11
The Writing Center at the University of Northern
British Columbia

• Formative-evaluation approach implemented during
  start-up.
• Followed by a small-scale evaluation.
• Focuses on one aspect of the program at a time.
• An objective-oriented evaluation with a summative
  emphasis was chosen.
• Determine whether the positive evaluations
  students gave to the center lasted.
• Determine whether students had learned something
  and gained long-term value.

12
Results

• 100 of clients were immediately satisfied with
  the objectives focused on in their conferences
  and they could immediately apply the newly gained
  knowledge to their school work.
• 83.3 agreed or agreed strongly that they were
  able to apply the knowledge gained after two
  weeks.
• 86.6 said that what they learned would be of
  benefit for the future.
• ¾ of the clients agreed that they could still
  apply the knowledge gained through the sessions,
  even after two months.
• The study was replicated, with new staff and
  tutors, and different student evaluator.
• Results were slightly less enthusiastic but still
  followed the same pattern as the previous
  evaluation.

13
What did the evaluation achieve?

• Confirmed that the Writing Center had a positive
  impact on students writing.
• A small-scale, objectives-oriented evaluation
  method works well for evaluating writing centers.
• There is value in evaluations
• Identities need
• Legitimizes the goals of the writing center
• Improves a centers chances long-term
  sustainability.

14
Questions or comments?

• Thank you.