PPA 502 Program Evaluation

1
PPA 502 Program Evaluation

• Lecture 6a Using Statistics Appropriately

2
Descriptive and Inferential Statistics

• Introduction
• Any phenomena that can be counted can be
  summarized. If these summaries are used to
  describe a group of items, the figures presented
  are descriptive statistics.
• When statistics are computed from a probability
  sample with the intention of generalizing from
  the sample to the population, the statistics are
  referred to as inferential statistics.

3
Descriptive and Inferential Statistics

• Generalizing from samples.
• The population of interest must be reasonably
  known and identifiable.
• A sampling technique should be used in which the
  probability for selecting any unit in the
  population can be calculated.
• A sample should be drawn that is of appropriate
  size relative to the size of the population to
  which generalization is desired.

4
Descriptive and Inferential Statistics

• Generalizing from samples.
• Even though probability sampling is applied,
  evaluators should examine a sample to ensure that
  it is truly representative of the population to
  which the evaluators hope to generalize.
• Without randomization, evaluators must take great
  care in assuring representativeness. With
  randomization, statistical significance is used.

5
Descriptive and Inferential Statistics

• Estimating the strength of relationships.

6
Descriptive and Inferential Statistics

• Statistical hypothesis testing.
• Null hypothesis vs. Research hypothesis.
• Discrepancies between true situation and test
  results.
• Type I error false positive.
• Type II error false negative.

7
Descriptive and Inferential Statistics

• Statistical hypothesis testing.

8
Descriptive and Inferential Statistics

• Statistical hypothesis testing.
• Selecting a statistical confidence test.
• 95 standard, but
• 80-90 may be more in line to avoid type II
  errors.
• Practical significance.
• Statistical significance measures whether
  findings can be generalized.
• Practical significance evaluates the size of
  program effect slight, moderate, strong.
• Unfortunately, no hard and fast standards.

9
Selecting Appropriate Statistics

• Criteria for selecting appropriate data analysis
  techniques.
• Question-related criteria.
• Generalization?
• Causal? Impact?
• Quantitative standards?

10
Selecting Appropriate Statistics

• Criteria for selecting appropriate data analysis
  techniques.
• Measurement-related criteria.
• Level of measurement?
• Multiple indicators?
• Sample sizes?
• Multiple observations over time?
• Independent or related samples?
• Variable distributions?
• Measurement precision?
• Outliers?

11
Selecting Appropriate Statistics

• Criteria for selecting appropriate data analysis
  techniques.
• Audience-related criteria.
• Audience knowledge of sophisticated techniques?
• Graphics versus tables?
• Precision level for audience?
• Graphs versus regressions?
• Statistical versus practical significance?

12
Selecting Appropriate Statistics
13
Selecting Appropriate Statistics

• Applying regression.
• Dependent variable.
• Linear model (but curvilinear can be modeled).
• Used to estimate changes in behavior or impacts.
• Best fitting line.
• Coefficient of determination.
• Unstandardized regression coefficients (slopes).
• Standardized regression coefficients (betas).
• Significance.
• T-test, individual.
• F-test, collective.
• Confidence intervals.

14
Selecting Appropriate Statistics

• Selecting techniques to sort measures or units.
• Techniques.
• Aggregation.
• Summative index.
• Analytical techniques.
• Measures.
• Factor analysis.
• Groups.
• Discriminant analysis.
• Cluster analysis.

15
Selecting Appropriate Statistics

• Other factors affecting selection of statistical
  techniques.
• Sample size.
• Number of observations over time.
• Variable distributions.
• Implied level of measurement.
• Outliers.
• Level of sophistication of users.

16
Selecting Appropriate Statistics

• Reporting statistics appropriately.
• Identify contents of all tables and figures
  clearly.
• Indicate use of decision rules in analysis.
• Consolidate analyses whenever possible.
• Do not abbreviate.
• Provide basic information about measurement of
  variables.
• Present appropriate percentages.
• Present information on statistical significance
  clearly.
• Present information on magnitude of relationships
  clearly.
• Use graphics to present analytical findings
  clearly.

17
Selecting Appropriate Statistics

• Reporting statistical results to high-level
  public officials.
• Dilemma how do you present less than certain
  data without excessive hedging.
• Prepare decision-makers for less than certain
  answers.
• Range of uncertainty (confidence intervals) are
  good because of familiarity with polling.
• Present only findings of practical importance.
• Graphics are better than tables.