What is Evaluation? - PowerPoint PPT Presentation

About This Presentation
Title:

What is Evaluation?

Description:

Title: Improving Learning and Teaching of English Through Evaluation, Measurement, and Assessment Author: David Williams Last modified by: A00946634 – PowerPoint PPT presentation

Number of Views:224
Avg rating:3.0/5.0
Slides: 26
Provided by: DavidW383
Category:

less

Transcript and Presenter's Notes

Title: What is Evaluation?


1
What is Evaluation?
  • David Dwayne WilliamsBrigham Young
    UniversityDavid_Williams_at_byu.edu

2
Evaluation, Assessment, Measurement, and Research
  • Evaluation usually includes describing
  • what is and
  • what should be, then
  • judging or comparing the two, as in a balance.
  • Measurement is an essential tool for gathering
    information about what is. John Brown example.
  • Assessments involve using measurement processes
    regularly for established purposes.
  • Research involves measuring what is, then seeking
    to understand and explain, not to judge.

3
Vocational Rehabilitation Examples
  • Utah Statewide ASSESSMENT of the Rehabilitation
    Needs of Individuals with Disabilities 2006-07
    Final Report.
  • Michael Leahys presentation yesterday on a
    synergistic program evaluation MODEL PARTNERSHIP
  • From the announcement of this conference
    consumer satisfaction studies, surveys, case
    file reviews, comprehensive needs assessments,
    economic impact studies, and use of other quality
    assurance measures.
  • Program Evaluation and Justification Review of
    the Rehabilitation Program Administered by the
    Department of Labor and Employment Security,
    Report No. 98-04, July 1998

4
Vocational Rehabilitation Evaluation
  • Has a long history in the literature (I found
    resources from the 1970s forward).
  • However, as in many fields, evaluations may turn
    out to be assessments, measures, or research
    rather than full evaluation.
  • Lets look at what the field of evaluation says
    about evaluation and then we can decide whether
    Vocational Rehabilitation might gain from what
    they offer

5
An Evaluation Framework based on Ideas From
Several Theorists
  • Alkin, 2004
  • Fetterman, 2001
  • Guba Lincoln, 1989
  • Patton, 2002, 2008
  • Stake, 2004
  • Stufflebeam, 2001, 2007
  • Weiss, 1998
  • Fitzpatrick, Sanders, Worthen, 2003

6
Evaluation Framework Overview
Evaluation Checklist
Program Evaluations Meta-Evaluation Checklist
  • Background Information
  • Audience Stakeholders
  • Evaluand Information
  • Stakeholder Concerns
  • Judging Criteria
  • Questions to Answer
  • Data Collection Processes
  • Data Analysis
  • Reporting Strategies
  • Results
  • Recommendations
  • Resource Valuation
  • Budget and Schedule
  • Self-Critique using meta-evaluation
  • Overview
  • Meeting Requirements for Utility
  • Meeting Requirements of Feasibility
  • Meeting Requirements for Propriety
  • Meeting Requirements for Accuracy

7
Context for understanding an Evaluation
  • What does the literature associated with the
    evaluand say are the key issues?
  • How did this evaluand come to be of interest to
    you?
  • What is your background that is relevant to this
    evaluation?
  • What evaluation has been done on this evaluand
    already?
  • Is the evaluand evaluable at present?
  • Why is an evaluation appropriate now?
  • What approaches to evaluation were considered and
    which will be used and why?

7
8
Possible VR Context Questions
  • What does the literature about Vocational
    Rehabilitation say ought to be included in a
    study?
  • How did this program or this counseling technique
    or this client come to be of interest to you?
  • What perspectives are you taking on this
    evaluation because of your particular background?
  • What might you be missing because of that?
  • What alternative views do you need to insist on
    including, besides your own?
  • What evaluation have you or others already done
    on this evaluand?
  • What has been learned from previous evaluations?

9
Who are the stakeholders who care? Why?
  • Who asked for the evaluation and why?
  • Who stands to benefit from the evaluation and
    how?
  • Who is served by the evaluand or should be?
  • Who is likely to use the evaluation results to do
    something helpful?
  • Who does not usually have a voice in matters
    associated with the evaluand but has a stake in
    it?

9
10
VR evaluators might ask about stakeholders
  • Who else besides me cares about this treatment,
    these resources, or this program?
  • Have any of them asked for an evaluation?
  • If so, why? If not, why not?
  • Why do I and these other people care about this
    program?
  • What do we stand to lose or gain by what happens
    with this program?
  • Who else is served by this program or should be
    and therefore should have an interest in its
    evaluation?
  • Are the administrators, other counselors, family
    members, employers, or others likely to use
    evaluation results to do something different?

11
What is the evaluand or thing the stakeholders
care about?
  • What do you already know about the evaluand?
  • What or Who it is
  • What its or their objectives are
  • How it works or what they are doing
  • What more do you need to learn to refine the
    description and definition of the evaluand so you
    can focus your evaluation on it or them?

11
12
What are the things or people VR evaluators
might evaluate?
  • One key evaluand may be them as counselors,
  • Or it may be the curriculum or program theyre
    using,
  • Or a particular technique they are piloting,
  • Their clients current performance, employment,
    concerns, and associated needs for improvement,
  • The relationships among several components of a
    program.
  • Or a test used to ascertain growth in client
    performance?

13
What criteria do stakeholders have for judging
the evaluand?
  • What values do the stakeholders manifest
    regarding the evaluand?
  • What do they think the evaluand should be
    accomplishing (criteria for success)?
  • What standards do they have or how completely do
    they hope the evaluand will meet the criteria?
  • How will they know when the evaluand is
    successful to their satisfaction?

13
14
VR evaluators might ask these criteria questions
  • What do we and other stakeholders value that
    should guide our evaluation efforts?
  • What should clients who participate in this
    program activity be able to do when they finish?
  • How well should clients perform on the selected
    criteria if the program is going to be considered
    successful?
  • What should counselors be doing, at what level of
    performance to help clients be successful?

15
What questions do stakeholders want to answer?
  • Based on the previous points, what evaluation
    questions should be asked?
  • Based on a rating or ranking of all possible
    questions raised, which are the highest
    priorities?
  • Which questions will this study address and why?

15
16
VR evaluators might ask these questions to match
the criteria
  • How are clients performing compared to the ideal?
  • Is there a need for an intervention change?
  • How well was the program implemented?
  • How many of the clients performed at or above 80
    on the job placement test?
  • How well did this counselor do in preparing their
    clients to apply for a job?
  • How well are we evaluating our interventions in
    terms of implementation and outcome?

17
What processes will be used to collect analyze
data to answer the questions compare the
evaluand to its criteria?
  • For each question listed earlier, what
  • Information will be collected analyzed?
  • Using what data collection procedures?
  • By whom and when?
  • How will each procedure be refined to ensure
    validity, reliability, credibility,
    trustworthiness, etc.

17
18
VR evaluators may collect analyze quantitative
or qualitative data by
  • Drawing upon formal measures developed by others
    or creating their own tests and performance
    activities,
  • Conducting informal interviews and observations,
  • Engaging clients in dialogues digitally
    recording them for analysis by the clients or
    others,
  • Analyzing these and other data both qualitatively
    quantitatively,
  • Comparing these descriptions of what is to the
    criteria and standards identified earlier.

19
What reporting recommendation strategies will
be used?
  • What interim reports will be given to whom and
    when?
  • What final reports will be given to whom and
    when?
  • How will the reports be organized, around what
    points?
  • Will there be oral reports? Written reports?
    Other formats?
  • How will results be organized and displayed?
  • What are the results or what results are
    anticipated?
  • Where will recommendations come from?
  • Will you be qualified to make recommendations and
    why?
  • What recommendations are there, who should
    implement them and how?

19
20
VR evaluators report results and recommendations
through the use of
  • Informal oral reports for own program
    evaluations,
  • Interim reports to share with others,
  • Formal written reports with charts and tables,
  • Reports on study progress stakeholder
    involvement,
  • Implications for future evaluation activities,
  • Evaluative judgments about quality of evaluands,
  • Realistic recommendations- using processes that
    involve the stakeholders who will implement the
    recommendations.

21
Metaevaluation of Evaluation Plans,
Implementation, Outcomes
  • Encourages high quality evaluations
  • Can be done internally or externally
  • Could involve Standards established by the Joint
    Committee of thoughtful professionals
  • Seeks to enhance evaluation quality in terms of
  • Utility
  • Feasibility
  • Propriety
  • Accuracy

22
VR evaluators should meta-evaluate to enhance
quality
  • When anticipating conducting an evaluation,
  • While conducting one, and/or
  • While reviewing evaluations performed by
    themselves or others.
  • Using Joint Committee standards to help clarify
    what they want to evaluate,
  • Using the Standards to judge how well they are
    evaluating,
  • By clarifying who they are serving with their
    evaluation, and how they value the results of
    their evaluation efforts.

23
Implications for VR Participants
  • Use Measurement and Assessment in a broader
    Evaluation context to enhance VR programs by
  • Attending to context, background and literature
  • Serving the values interests of all
    stakeholders
  • Involving stakeholders in clarifying the
    evaluand, criteria, standards they care most
    about
  • Targeting stakeholders questions with a variety
    of data collection and analysis methods that
    involve measures of high quality to assess how
    well what is matches up with what should be
    for the stakeholders
  • Sharing results and recommendations that are
    realistic and useful for the stakeholders in ways
    they can use.

24
Come Learn More This Afternoon at a Workshop. We
will
  • Review the evaluation framework presented here
  • Discuss the premise that measurement and
    assessment are means for doing evaluation and
    research
  • Discuss and write down current practices and
    questions about evaluating your work activities.
  • Develop plans for applying these ideas to your
    practice
  • Share emerging plans with other participants for
    feedback
  • Receive guidance and feedback from presenter
  • Accept the challenge to apply this plan at home
    and to contact the presenter with questions and
    further guidance if wanted.

25
For more information or questions,
  • Contact
  • David Williams
  • 150 G MCKB
  • Brigham Young University
  • Provo, UT 84602 USA
  • David_Williams_at_byu.edu
Write a Comment
User Comments (0)
About PowerShow.com