S519: Evaluation of Information Systems - PowerPoint PPT Presentation

Loading...

PPT – S519: Evaluation of Information Systems PowerPoint presentation | free to download - id: 78d724-NjE2N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

S519: Evaluation of Information Systems

Description:

S519: Evaluation of Information Systems Analyzing data: value and importance Ch6+7 Last week How to infer causation: 8 strategies? How to put them together? – PowerPoint PPT presentation

Number of Views:26
Avg rating:3.0/5.0
Slides: 29
Provided by: nori93
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: S519: Evaluation of Information Systems


1
S519 Evaluation of Information Systems
  • Analyzing data
  • value and importance
  • Ch67

2
Last week
  • How to infer causation 8 strategies?
  • How to put them together?

3
Values in evaluation (DCh6)
  • Adding value to descriptive data to make our
    evaluation explicit
  • Our goal
  • Using quantitative value to evaluate the quality
    or value of the evaluand in a particular context.
  • Build up our conclusions based on a level of
    certainty
  • What are values
  • good, valuable, worthwhile

4
Values in evaluation
  • Adding values to descriptive data collected
    about
  • Process, outcomes, costs, comparisons,
    exportabilities
  • Weighting all the strengthens or weaknesses of
    these values to draw overall conclusion about the
    evaluand. How
  • Importance weighting
  • Merit determination
  • synthesis

methodology
5
Subjective
  • Before we go for methodology, we have to answer
    the question
  • Whether our data are subjective

6
Three types of subjectives (Scriven, 1991)
  • Subjective 1 Inappropriate application of
    personal or cultural preferences/biases
  • arbitrary, idiosyncratic, unreliable, highly
    personal (i.e., based purely on personal
    preference, cultural biases, gender biases)
  • Subjective 2 assessment or interpretation by a
    person, rather than a machine, of something
    external to the person
  • Using well-founded expert judgements
  • Robust evidence
  • Subjective 3 about a persons inner life or
    experiences (e.g., headaches, fears, beliefs,
    emotions, stress)
  • Usually not independently verifiable

7
Avoid Subjective
  • We provide our conclusion baesed on certainty in
    the relevant decision-making context
  • Keep the whole evaluation well documented and
    justified
  • All evaluations, especially high-stakes ones,
    should be meta-evaluated (i.e., evaluation itself
    should be evaluated)

8
Exercise
  • Suppose that a client does not like the findings
    of your evaluation and says
  • Well, that is just your opinion
  • evaluations are always just so subjective
  • How could you response?
  • Form a group and discuss

9
Exercise
  • Do you agree or disagree
  • All evaluative claims (about the value of certain
    outcomes or attributes, their relative
    importance, and what mixed results indicate about
    overall value) are arrivated at subjectively
    more specifically, the values that are applied to
    descriptive facts (data) to arrive at evaluative
    conclusions are personal values.
  • Allowing stakeholders to make up their own minds,
    either individually or collectively, is the only
    valid way in which evaluative conclusions can be
    drawn at all.
  • Form a group with the same opinion and debates
    with another group holding opposite opinion.

10
Determining importance (D-ch7)
  • Importance determiniation is the process of
    assigning labels to dimensions or components to
    indicate their importance.
  • Importance weighting
  • Prioritize improvements
  • Identify whether identified strengths or weakness
    are serious or minor
  • Work out whether an evaluand with mixed results
    is doing fairly well, quite poorly, or somewhere
    in between.

11
Determining importance (D-ch7)
  • Different evaluations
  • Dimensional evaluation
  • Looking at multiple dimensions of merit that
    pertain to the evaluand as a whole rather than
    separately to its parts.
  • Component evaluation
  • Looking at each of the evaluands components (or
    parts) separately and then synthesizing these
    findings to draw conclusion about the evaluand as
    a whole.
  • Each component can be evaluated on several
    dimensions that pertain to this component only
    rather than to the evaluand as a whole.
  • Holistic evaluation
  • Looking evaluation as a whole without division
    into dimensions or components

12
Determining the importance of dimensions
13
Determining the importance of componentss
14
Determining importance
  • Weak performance on minor criteria (e.g.
    dimensions, components) may be no big deal,
  • But weak performance on important criteria can be
    very serious issues.

15
When to use what
  • Component analysis
  • Evaluating policies, programs, or interventions
    that have several quite distinct parts
  • An international program consisting of projects
    implemented in different locations (e.g. WIC in
    IU)
  • A government policy includes multiple policy
    measurements (e.g. Juvenile delinquency)
  • An organizational transformation includes several
    distinct interventions (e.g. Career support)

16
When to use what
  • Dimensional evaluation
  • Entities whose quality or value is experienced by
    consumers on multiple dimensions that pertain to
    the evaluand as a whole
  • Single-component program or intervention
  • Product evaluation (i.e. Car evaluation)

17
When to use what
  • Holistic evaluation
  • Unusual in the evaluation of programs, policies
    and other large complex evaluands.
  • More common in personnel, product and service
    evaluation (expertise-oriented evaluation)
  • Judging the overall quality of a sample of
    writing
  • Grading essays
  • Classroom teaching
  • Athletic performance
  • cosmetics

18
Determining importance 6 strategies
  • 1. having stakeholders or consumers vote on
    importance
  • Commonly used in both participatory and
    nonparticipatory evaluations
  • Collecting opinions from everybody
  • Assumptions
  • Each person is well informed
  • Stakeholders belief what (s)he chooses is
    important
  • Stakeholders important should be treated equally
  • Pros and cons?

19
Determining importance 6 strategies
  • 2. Drawing on the knowledge of selected
    stakeholders
  • Using selelcted stakeholder input to guide the
    assignment of importance weightings
  • Collecting opinions from selected experts
  • Setting up the Bars
  • A bar is a defined minimum level of critierion
    performance below which the evaluand is
    considered completely unacceptable, regardless of
    performance on other criteria.

20
Determining importance 6 strategies
  • 2. Drawing on the knowledge of selected
    stakeholders
  • Assumptions
  • The stakeholders should be sufficiently well
    informed to provide valuable relevant information
  • The combination of stakeholder input wil provide
    sufficient certainty about importance for the
    given decision-making context
  • Prons and cons?

21
Determining importance 6 strategies
  • 3. Using evidence from the literature
  • Literature review
  • Evaluations of similar evaluations in similar
    contexts
  • Research documenting the key drivers (or
    strongest predicators) of success or failure with
    this type of evaluand.
  • Assumptions
  • The volume and quality of the available research
    is sufficient to judge the importance
  • The context of other research is sufficiently
    similiar to yours and therefore that the findings
    can be reasonably applied to your setting
  • Prons and cons?

22
Determining importance 6 strategies
  • 4. Using specialist judgment
  • When you have tight timeline, no time for
    gathering stakeholders and looking for literature
  • Identify one or two (or two or more) well-known
    specialists in the domain
  • Better be supplemented with other evidence
  • Prons and cons?

23
Determining importance 6 strategies
  • 5. Using evidence from the needs and values
    assessments
  • Determining the importance of criteria
    (dimensions)
  • Still remember needs and value assessment (see
    Table 3.3)?
  • How to get importance from Table 3.3?
  • Any frequently mentioned characteristics?
  • Looking for poor-performing evaluators that cause
    serious problem
  • Looking for top-notch evaluators that have
    dramatic impacts on success

24
Determining importance 6 strategies
  • 5. Using evidence from the needs and values
    assessments
  • Determining the importance of components
  • Severity of dysfunction addressed (primary
    consideration)
  • Scarcity of alternatives no other options for
    addressing the need.
  • Intent to use alternatives if the evaluand
    component in question did not exist.
  • Rubics to measure (Table7.4, 7.5 and 7.6, 7.7,
    7.8 (combined))
  • Prons and cons?

25
Exercise
  • Join your project group
  • Discuss Table 7.8
  • Think about how to draw similar table for your
    group project

26
Determining importance 6 strategies
  • 6. Using program theory and evidence of causal
    linkages
  • When criteria or components are linked to needs
    through a complex logic chain.
  • Such as soft skills or attributes (e.g.,
    inspirational leadership, self-esteem, stress
    management, a kind of instrumental needs)
  • More upstream variables (see Exhibit 7.5)
  • How to estimate the strengths of the links
  • Interview
  • Anaylse your previous data
  • ...

27
Strengths and weaknesses of the six strategies
  • Always think whether they are applicable
  • Choose mulitple of them

28
Exercise
  • Table 7.10 (class dissusion)
  • Form a group
  • Discuss which strategies you will choose to
    determine the importance for the student
    services in the school health program (see Table
    7.8)
  • Discuss which strategies you will choose to
    determine the importance for your group project
About PowerShow.com