Dr. Kimberly Kempf-Leonard - PowerPoint PPT Presentation

Loading...

PPT – Dr. Kimberly Kempf-Leonard PowerPoint presentation | free to download - id: 4a57e4-NjM1O



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Dr. Kimberly Kempf-Leonard

Description:

Methods & Sources of Data: How, Who & What Subcontractors should Observe Presented by: Dr. Kimberly Kempf-Leonard School of Social Sciences University of Texas at Dallas – PowerPoint PPT presentation

Number of Views:47
Avg rating:3.0/5.0
Slides: 32
Provided by: mue90
Learn more at: http://dsgonline.com
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Dr. Kimberly Kempf-Leonard


1
Methods Sources of Data How, Who What
Subcontractors should Observe
  • Presented by
  • Dr. Kimberly Kempf-Leonard
  • School of Social Sciences
  • University of Texas at Dallas
  • kleonard_at_utdallas.edu

Performance Measurement Training
2
Whats it mean to be real?
  • Is it real in eye of beholder?
  • Subjective reality
  • Is it the same reality for everyone?
  • Objective reality
  • Goal Real Information on Outcomes
  • distinguish objective subjective

3
We know how to lose weight
  • 1980-90s Hypothesis Weight loss a longer life
    are both more likely when you eat a low fat diet.

  • FAT
    gtgtgtgtCAUSESgtgtgtgt FAT
  • Fat is bad for you it makes sense
  • pasta, bagels, bran muffins
  • U.S. Food Pyramid
  • 2000s Hypothesis Weight loss and a longer life
    are both more likely when you eat a diet low in
    carbohydrates.

  • CARBS gtgt CAUSE gtgtFAT
  • Atkins diet Red Meat
  • Endocrinology 101 theory carbs gt
    insulin/blood sugargtmetabolize fatgtappetite
  • Epidemic of obesity and Type II diabetes
    clinical trials low fat diet fails
  • 2005 Hypothesis There isnt just 1 right
    answer
  • Move More Eat Less X, Y X gtgt
    CAUSE gtgt FAT

4
How do you know .(X)..?
  • X is common sense or obvious
  • X is a generally agreed upon belief
  • X is based on tradition
  • Experts told me X
  • I learned X through my own experience
  • I observed X
  • Could you be wrong?
  • Inaccurate observations
  • Overgeneralization
  • Selective Observations
  • Illogical reasoning
  • Resist change
  • Ego-based commitments
  • too devoted to tradition
  • Uncritical agreement with authority

5
Quality Control Standards.. because we dont use
the same vocabulary, have the same experiences,
and interpret in the same way.
  • Precision level of specificity desired
  • Reliability consistency dependability
  • Validity accuracy
  • Generalizablity wider applicability
  • Objectivity not subjective
  • Control measurement error Random systematic
    error
  • Cross-check (2 eyes are better than 1)

6
METHODS of Data CollectionAsk...WatchExperience
Use whats there
  • Archival records
  • Documents, Information Systems, Content
  • Field Observations
  • Personal Experiences
  • Surveys
  • Questionnaires, Interviews, Focus Groups

7
Frameworks or Designs
  • Cross-sectional designs
  • Data are collected for a single point in time
  • Quickest
  • Only correlations, cant assess cause/effect
  • Longitudinal designs
  • Data are collected for at least two points in
    time
  • Only way really to assure cause and effect
  • Case Study
  • Single case serves as the whole subject
    (sublevels)
  • Provides rich detail
  • Difficult to generalize to other cases

8
For which youths should they get data? (subjects)
  • Good answers
  • The population
  • A sample that represents the population well

9
Representativeness Generalizability
10
How well does the sample represent the population?
11
Selecting a sample
  • Probability techniques (random choices)
  • Simple random
  • Stratified random
  • Multistage or cluster random
  • Non-probability techniques
  • Convenient or happenstance
  • Quota or example

12
How many subjects do we need?(sample size)
  • Selection is critical for unbiased estimates, not
    size
  • margin for error calculation only possible
    with probability selection
  • Size
  • Front end select to represent population
  • Back end minimal attrition bias
  • 25-50 cases adequate for small sample analysis
  • Larger samples more breadth less depth
  • Want larger if targeted behavior or outcome is
    infrequent

13
What sort of information is recorded? (variables)
  • Output measures of Program Activities
  • hours of training
  • of youth served
  • Average days in program services
  • of services implemented
  • Average days from intake to disposition
  • Outcome measures (Short- Long-term)
  • of youth who reoffend
  • of youth who complete program requirements

14
Outputs Outcomes
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
A variable should try to hit
19
What do we know about unemployment rate.
  • U.S. unemployment rate is measured by dividing
    the number of unemployed individuals by the
    number of people in the civilian labor force
    multiplying by 100
  • Civilian Labor force exclusions military,
    inmates, ltage 16, homemakers, students,
    retirees, disabled, voluntarily idle.
  • Unemployed individuals include working at paid
    job (inc. part-time), unpaid work at family
    business, absent from work (paid unpaid) due to
    labor dispute, bad weather, vacations, personal
    reason

20
more confidence crime declined
21
  • What happened to the exit polls in the 2004
    Presidential election?
  • What does it mean for a youth to re-offend or
    recidivate?
  • What does it mean to participate in the program?
  • Is there any type of youth not included? Does
    that make a difference?
  • What does if mean if theres a handwritten A in
    the blank for race?

22
Assessing variables
  • Characteristics
  • Multiple categories
  • Mutually exclusive
  • Exhaustive
  • Seldom perfect measures
  • Fail to get true meaning equate measure proxy
  • Random mistakes
  • Systematic mistakes
  • Multiple measures improve confidence
  • 2 eyes are better than 1

23
Survey Methods
  • Standardized, systematic asking
  • Uniform stimulus so responses can be compared
  • Efficient data collection
  • Price, Speed, Quality Pick Two
  • Ideal for characteristics of a population
  • Individuals attitudes, values, behaviors,
    experiences, opinions, knowledge, circumstances
  • Organizations/Institutions culture, policies,
    finances

24
The heart of surveys is the questionnaire
instrument
25
Anticipate how people might answer
  • Interpretation of the Question
  • Retrieve from memory
  • Form judgement
  • Edit response for social desirability (strategic)
  • Trying out (pretesting) survey questions helps
    avoid problems

26
They Need to be Clear
  • Avoid extreme loaded positions
  • Should State spend more money on youth services?
  • Should State spend more money to help serious
    juvenile offenders?
  • Limit to a single topic (Double-barreled
    Conditional phrases)
  • Do you think your outlook and performance
    benefited from the curriculum and the teachers?
  • Avoid ambiguity
  • The number of youths living here most of the
    time while working, even if they have another
    place to live.
  • Complexity creates confusion
  • Im going to read you a list of race categories.
    Please choose one or more categories that best
    indicate your race.
  • Avoid Presupposition
  • What are your usual practices for studying?
  • Varied verb status (have you stopped attending
    a program?)
  • Forced choice (Are you truant or delinquent?)

27
Measurement error in surveys
  • Sampling (frame, selection, statistical
    inference)
  • Collection (method, instrument, interviewer,
    respondent)
  • processing (coding, data entry, data transfer,
    documentation)
  • Random or systematic
  • Systematic bias/error alters estimates

28
(No Transcript)
29
OUTCOME MEASURES
OUTPUT MEASURES
ACTIVITIES
SHORT-TERM
LONG-TERM
/ of program youth exhibiting desired change in
targeted behaviors (e.g., substance use, school
attendance, antisocial behavior, family
relationships, pregnancies)
/ of program youth exhibiting desired change in
targeted behaviors (e.g., substance use, school
attendance, antisocial behavior, family
relationships, pregnancies)
Conduct Planning Activities Implement Program(s)
/ of program youth completing program
requirements
/ of youth satisfied with program
/ of program families satisfied with program
Monitor Program(s)
/ of program staff with increased knowledge of
program area
Performance Measures should report on those
activities funded by Title II (Formula Grants)
funds.
Outcome Measure Definitions Short-TermOccurs
during the program or by the end of the
program Long-TermOccurs 6 months to 1 year after
program completion
30
(No Transcript)
31
  • Precision Does it achieve the level of
    generality intended?
  • Validity Is it accurate?
  • Does it capture reality of
    whats intended?
  • Reliability Do repeated tries yield the same
    result?
  • Assess comparisons cross-checks
  • Do both eyes see it the same way?
  • Use established measures
  • data collection instruments
  • cross-check data collectors
  • test-retest method (2 different times)
  • split-half method (1/2 one kind ½ another)
About PowerShow.com