S519: Evaluation of Information Systems - PowerPoint PPT Presentation

Loading...

PPT – S519: Evaluation of Information Systems PowerPoint presentation | free to download - id: 7b6727-NmQ3Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

S519: Evaluation of Information Systems

Description:

Title: L643: Evaluation of Information Systems Author: noriko Last modified by: ying Created Date: 1/6/2002 11:37:21 PM Document presentation format – PowerPoint PPT presentation

Number of Views:52
Avg rating:3.0/5.0
Slides: 30
Provided by: nori99
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: S519: Evaluation of Information Systems


1
S519 Evaluation of Information Systems
  • Analyzing data
  • Synthesis
  • D-Ch9

2
Synthesis methodology
  • It is a tool to allow us to draw overall
    evaluative conclusions from multiple findings
    about a single evaluand.
  • Synthesis is the process of combining a set of
    ratings or performances on several components or
    dimensions into an overall rating.

3
Synthesis methodology
  • Merit determination
  • To develop the rubrics
  • To use rubrics to summarize the multiple findings
  • Rubrics are one of the simplest methods to blend
    data.
  • But when data is a bit more complex, it is
    difficult to use a rubric as the only tool
  • Data are not equally important or reliable
  • Multi dimensions or multi components
  • Different nuances and combinations (such as
    Table8.3)

4
It is not
  • It is not meta-analysis
  • A special statistical technique to give a
    weighted average of effect sizes across multiple
    studies for quantitative studies
  • It is not literature review or a summary
  • A judgment from a reviewers point of view.

5
Keep in mind
  • Doing poorly on some minimal important criteria
  • Doing poorly on some crucial criteria
  • Are very different!

6
Evaluation
  • Synthesis for ranking
  • If it is ranking (relative) evaluation
  • Consider each alternative and make explicit
    comparisons
  • Synthesis for grading
  • If it is grading (absolute) evaluation
  • Consider different context settings and provide
    better interpretation of merit

7
Qualitative or quantitative
  • Quantitative synthesis
  • Using numerical weights
  • Qualitative synthesis
  • Using qualitative labels

8
Synthesis for grading
  • The primary evaluation question is for absolute
    quality or value
  • How well did the evaluand perform on this
    dimension or component?
  • How effective, valuable, or meritorious is the
    evaluand overall?
  • Is this worth the resources put into it?

9
Quantitative weighting example with bars
  • Case Personnel evaluation in a small accounting
    firm
  • 13 defined tasks (e.g., telephone, reception,
    data entry, etc.)
  • Each employee has responsibility for 4-6 tasks
  • Evaluation
  • Importance weighting (through the voting of the
    selected stakeholders)
  • In-depth discussion with business owners
  • Derive the importance metric and bars

10
Quantitative weighting example with bars
  • Evaluation
  • Define the levels of importance
  • 3 to 5 levels work well in most case
  • Do not go to too many levels (why? Is this
    useful?)
  • For example
  • task
  • 1. minor task (1)
  • 2. normal-priority task (2)
  • 3. high-priority task (3)
  • 4. extremely high-priority task (4)

11
Quantitative weighting example with bars
  • Evaluation
  • Setting up rubrics for each 13 tasks
  • Normally 4-6 level is sufficient
  • Example Performance Rubric
  • 1. Totally unacceptable performance (1)
  • 2. Mediocre (substandard) performance (2)
  • 3. Good performance (expected level) (3)
  • 4. Performance that exceeded expectations (4)
  • 5. All-around excellent performance (5)
  • Synthesis draw the overall conclusion
  • See Exhibit 9.2 (p158)

12
Exercise
Lab
  • Personal evaluation in a small accouting firm

Tasks Importance Score for Alice
Telephone 1 2
Data entry 2 3
Tax data management 4 1
Client support 4 5
Reporting 3 3
Communicating 1 3
How about Alice according to Exhibit 9.2?
13
Exercise
Lab
  • Personal evaluation in a small accouting firm

Tasks Importance Score for John
Telephone 1 2
Data entry 2 3
Tax data management 4 2
Client support 4 5
Reporting 3 3
Communicating 1 3
How about John according to Exhibit 9.2?
14
Exercise
Lab
  • Perosnal evaluation in a small accouting firm

Tasks Importance Score for Chris
Telephone 1 2
Data entry 2 3
Tax data management 4 4
Client support 4 5
Reporting 3 3
Communicating 1 3
How about Chris according to Exhibit 9.2?
15
Exerice
  • How about Chris
  • Mean 122344453313/(124431) 56/1
    5 3.73
  • What is Chris performance?

16
Qualitative weighting example 1 (with no bars)
  • Case a school-based health program evaluation
  • It contains 9 different components nutrition
    education, mental health services, safer sex,
    legal service and others.
  • How to evaluate these systems in low-budget and
    short period of time whether they are meeting
    important needs of the students and their
    families
  • Evaluation
  • Interview
  • Student surveys

17
School health system evaluation
  • Survey question design
  • Two quantitative questions
  • How useful was the program to you? (4-point
    response scale not at all useful, somewhat
    useful, useful, very useful)
  • How satisfied were you with the program?
  • One qualitative question (open-end)?
  • What other changes or events, good or bad, have
    happened to you or someone you know because of
    receiving the service?

18
School health system evaluation
  • Survey result about nutrition system shows in
    Table 9.1
  • Look at table 9.1, think about
  • How can you draw a conclusion from this result
    about the nutrition system? Is it good or bad?

19
School health system evaluation
  • Setting the importance for these three questions
    (1-strongest data, 3weakest data)
  • 1. Ratings of usefulness (directly related to
    needs)
  • 2. Responses to the open-ended question
  • 3. Satisfaction ratings
  • Creating rubrics for each question
  • Table 9.2 for question 1 and question 2
  • Table 9.3 for open-ended question

20
School health system evaluation
  • How to grade the nutrition system based on the
    first two quantitative questions
  • Based on Table 9.1, come out with the rubric as
    Table 9.2
  • Why 90 is select, 70-90..
  • How to draw Table 9.2 from Table 9.1 and
    collected data?

21
School health system evaluation
  • Table 9.3
  • Rubric for converting data from qualitative
    evaluation - open-ended responses into merit
    ratings
  • Is that a good way to do this?
  • Are you happy with this table?
  • If not, how do you want to improve it?

22
School health system evaluation
  • Synthesis to draw overall conclusion
  • Step-by-step
  • Start with the strongest data (question 1)
  • Blend with open-ended comments
  • Finally take the satisfaction ratings into
    account
  • See table 9.4 for the whole process

23
School health system evaluation
  • How to draw final conclusion?

Usefulness ratings
Final coclusion Merit of the nutrition program
Satisfaction ratings
Open-ended comments
Using quantitative ratings to draw the suggested
results and using qualitative ratings to find the
positive or negative facts to re-adjust the
results
See table 9.4 Discuss how to apply this to your
group project
24
Qualitative (nonnumerical) weighting example 2
  • Bar
  • A minimum level of performance on a specific
    dimension
  • Performance below this cannot be compensated for
    by much better performance on other dimensions
    (see Exhibit 9.2)
  • Hard hurdle (also referred as global bars)
  • Overall passing requirement for an evaluand as a
    whole (see Exhibit 9.2)
  • Soft hurdle
  • Overall requirement for entry into a high rating
    category
  • Place a limit on the maximum rating (e.g., I want
    all As for my classes)

25
Qualitative (nonnumerical) weighting example 2
  • Case Evaluation of the learning capacity of a
    small biotechnology start-up company biosleep.
  • Evaluation
  • 27 subdimensions of organizational learning
    capacity (see table 9.5)
  • Data collection survey and interview
  • Rubric similar as Table 8.2
  • Importance is built by using strategy 6 in
    Chapter 7
  • Using program theory and evidence of causal
    linkages (p118-125)

26
Biosleep
  • Evaluation
  • Synthesis
  • Pack the ratings on the subdimensions into 8 main
    dimensions
  • Combine the ratings on these 8 main dimensions to
    draw an overall conclusion

27
Biosleep
  • Dimension by dimension
  • Layer by layer

Sub-dimnention1
Dimnention1
Sub-dimnention2
Overall rating
Sub-dimnention3
Dimnention2
Sub-dimnention4
28
Biosleep
  • Synthesis
  • Subdimensions ? Dimensions
  • Using Table 9.6 to draw conclusions of dimentions
    based on subdimensions
  • Using Table 9.6 to judge Table 9.5 and come out
    the result as Exhibit 9.4
  • Dimensions ? overall evaluation
  • Based on Table 9.7 (created based on literature
    review,
  • What is your conclusion for the evaluation of
    Biosleep? And why?

29
Exericse
Lab
  • Form your group project
  • Discuss on how are you going to grade your
    evaluation?
  • Which example you would like to follow?
  • How to develop rubric for dimension and overall?
About PowerShow.com