THE QUANTITATIVE/ QUALITATIVE IMPACT STUDY

1
THE QUANTITATIVE/QUALITATIVE IMPACT STUDY

• BENEFITS AND CHALLENGES
• Dr. Gloria Pryor James
• Dr. Brenda R. Shaw
• Mr. Samuel T. Rhoades, J. D., President Emeritus
• National Association of HBCU Title III
  Administrators, Inc. Technical Assistance
  Workshop
• June 23, 2011

2
WHAT IS AN IMPACT STUDY?

• The Program Impact Study is one of the most
  important processes that identifies over time the
  impact of the Institutions programmatic outputs
  and inputs and the impact that these components
  had on the target population and the University,
  as a whole.

3
SOME BENEFITS OF THE QUANTITATIVE/QUALITATIVE
IMPACT STUDY

• Specific outcomes can be compared to resources
  expended and successful programs can be
  highlighted

4
SOME BENEFITS (CONTINUED)

• Programs that have not accomplished their
  intended outcomes can be identified
• The extent to which the planned outcomes were met
  can be documented

5
SOME BENEFITS (CONTINUED)

• The adequacy of the programs outcomes as related
  to the problem that the program was developed and
  implemented to address can be documented

6
SOME BENEFITS (CONTINUED)

• Strengths and weaknesses of each activity can be
  identified that affected the Effectiveness and
  Adequacy of the programmatic inputs (, human
  resources, space, etc.) in meeting the activitys
  planned outcomes.

7
SOME BENEFITS (CONTINUED)

• The institution has the ability to determine the
  impact of the Title III funds awarded to various
  discrete activities and programs.

8
SOME BENEFITS (CONTINUED)

• The process determines not only the success or
  failure of the program, but also provides
  explanations for the actual outcomes.

9
PRIMARY BENEFIT

• A COMPREHENSIVE picture of the overall impact of
  the programs being evaluated is the major
  outcome.

10
WHY CONDUCT EVALUATIONS AND IMPACT STUDIES

• As a result of the MANDATE of funding agencies
  and institutions to demonstrate programmatic
  accountability, one of the services requested
  most by institutions of higher education is
  program ASSESSMENT/EVALUATION.

11
(No Transcript)
12
ABOUT NORTH CAROLINA CENTRAL UNIVERSITY

• NCCU, chartered in 1909, opened its doors to
  students in 1910 as the National Religious
  Training School and Chautauqua.

13
ABOUT NORTH CAROLINA CENTRAL UNIVERSITY

• North Carolina Central University is a part of
  the University of North Carolina System, made up
  of 16 individual campuses (17 when we count the
  School of the Arts) and headed by a single
  president and governed by the University of North
  Carolina Board of Governors.
•  

14
ABOUT NORTH CAROLINA CENTRAL UNIVERSITY

• North Carolina Central University is a
  comprehensive university offering bachelors
  degrees in more than 100 disciplines, masters
  degrees in more than 40 programs and three
  professional degrees in law.

15
ABOUT NORTH CAROLINA CENTRAL UNIVERSITY

• Enrollment at North Carolina Central University
  increased from 5,753 in Fall 2001 to 8, 619 for
  Fall 2006, an increase over the six year period
  of 33. We currently enroll 8,500 students
  (2010-2011).
•  
• Major emphasis for the current planning period
  (2007-2012) is Enhancing Retention, Graduation,
  and Placement Rates through Increased Student
  Learning.

16
ABOUT NORTH CAROLINA CENTRAL UNIVERSITY

• Our goals
• to provide programs, activities, and services
  that will assist students in becoming successful
  during their tenure at the University so that
  they persist and graduate and are successful in
  the career markets of the future
• to put in place those systems designed to make
  the matriculation more seamless and easier to
  negotiate

17
ABOUT NORTH CAROLINA CENTRAL UNIVERSITY

• to provide the training and upgrading of the
  skills of our faculty to work with those students
  whom we enroll that have been underserved and are
  from diverse backgrounds and
• to promote a more efficient, effective, and
  accountable environment as relates to student
  learning and student success.

18
INCEPTION OF THE IMPACT STUDY INTERNAL
MOTIVATION

• Impetus for conducting an Impact Study
  Chancellor Charlie Nelms
• A directive he gave me as Title III Director,
  which I passed on to the Activity Coordinators in
  our planning meeting
• His absolute need to have concrete documentation
  of the impact that our Title III dollars had had
  over the years.

19
CONTRACT OUT OR CONDUCT INTERNALLY

• North Carolina Central University (NCCU)
  contracted with Associates for
  Institutional Development, Inc. (AID, Inc.) to
  conduct an Impact Study of select Title III
  activities encompassing the grant years 2004-2009
  to document the extent to which each activity had
  accomplished the outcomes for which the activity
  was funded.

20
THE KEY IS COLLABORATION

• THE RESEARCH TEAM MUST

• THE INSTITUTION MUST

• PROVIDE ACCESS TO ALL RECORDS PERTINENT TO
  EACH ACTIVITY

• WORK CLOSELY WITH THE TITLE III STAFF TO COLLECT
  THE DATA

21
ASSESSMENT GOALS OF THE IMPACT STUDY

• A data collection process MUST be implemented
  that includes
• Onsite visits
• Interviews with key staff
• Participant self-assessments

22
ASSESSMENT GOALS OF THE IMPACT STUDY

• Interviews with faculty, students, and external
  stakeholders
• The administration of surveys and questionnaires,
  as needed.

23
ASSESSMENT GOALS OF THE IMPACT STUDY

• To guarantee objectivity, the process relies upon
  multiple sources of data encompassing multiple
  points of view. Finally, to be adaptable, the
  assessments design must meet the assessment
  needs of the institution.

24
INFORMATION NEEDED FOR THE STUDY

• PROGRAM PLANS (RATIONALE, TARGET AUDIENCE,
  BASELINE DATA
• MEASURABLE OBJECTIVES
• MEASURABLE AND TIME-SPECIFIC PERFORMANCE
  INDICATORS
• IMPLEMENTATION STRATEGIES IN CHRONOLOGICAL ORDER
•  

25
INFORMATION NEEDED FOR THE STUDY

• DOCUMENTATION ON ACTUAL PROGRAMMATIC OUTCOMES
  (DESCRIPTIVE AND STATISTICAL)
• EXTERNAL EVALUATION REPORTS
• PROGRESS REPORTS (MONTHLY OR QUARTERLY)
•  
•  

26
INFORMATION NEEDED FOR THE STUDY

• PROGRAMMATIC OUTCOMES
• (QUANTIFIABLE AND
• DESCRIPTIVE)

27
INFORMATION NEEDED FOR THE STUDY

• ANNUAL REPORTS
• RESULTS OF SURVEYS AND QUESTIONNAIRES
• FUNDS AWARDED EACH ACTIVITY DURING THE YEARS
  INCLUDED IN THE STUDY 

28
INFORMATION NEEDED FOR THE STUDY

• ACTIVITY BUDGET BURN RATES
• ANEDOTAL INFORMATION THAT WILL ADD VALUE TO THE
  OUTCOMES

29
CRITICAL CHALLENGES TO CONDUCTING AN IMPACT STUDY

• OBJECTIVES THAT ARE NOT MEASURABLE
• TOO MANY TASKS MASQUERADING AS OBJECTIVES
• PERFORMANCE INDICATORS THAT ARE NOT MEASURABLE
  AND NOT TIME-SPECIFIC

30
CRITICAL CHALLENGES TO CONDUCTING AN IMPACT STUDY

• THE ABSENCE OF BASELINE DATA FOR THE FIRST YEAR
  OF IMPLEMENTATION AND EACH YEAR OF FUNDING (WHERE
  APPROPRIATE)
• LACK OF ANNUAL EVALUATION REPORTS OR SOME OTHER
  TYPE OF ASSESSMENT TO VERIFY THE ATTAINMENT OF
  THE OBJECTIVES EACH YEAR OF FUNDING

31
CRITICAL CHALLENGES TO CONDUCTING AN IMPACT STUDY

• ACTIVITY DIRECTORS NOT HAVING A CLEAR VISION OF
  THE PRIMARY ANTICIPATED/PLANNED OUTCOMES OF THE
  ACTIVITY AND HOW THE ACTIVITY WILL STRENGTHEN THE
  INSTITUTION

32
IMPACT STUDY PROCESS USED AT NCCU

• Identified the over-arching goal or planned
  outcome for each program
• Identified the actual outcome based on data

33
IMPACT STUDY PROCESS USED AT NCCU

• Analyzed the gap between the desired outcome and
  the actual outcome by utilizing the
  Effectiveness Estimate (EE) technique

34
EFFECTIVENESS ESTIMATE (EE) TECHNIQUE

• DOCUMENTS
• THE EFFECTIVENESS OF THE PROGRAM IN AMELIORATING
  OR STRENGTHENING THE ENTITY FOR WHICH IT WAS
  FUNDED

35
ELEMENTS OF THE EFFECTIVENESS ESTIMATE

• (R) Actual Results
• (C) Results Without Program
• (P) Planned Impact

36
THE EFFECTIVENESS ESTIMATE

• EFFECTIVENESS ESTIMATE
• (R) Actual Results - (C) Results Without Program
• Divided By
• (P) Planned Impact - (C) Results Without Program
  

37
ADEQUACY RATIO (AR) 

• QUANTIFIES
• PROGRAM ACCOMPLISHMENT BY CONCEPTUALIZING THE
  PROPORTION OF THE PROBLEM ELIMINATED BY THE
  PROGRAM

38
ELEMENTS OF THE ADEQUACY RATIO

• (R) Actual Results
• (C) Results Without Program

39
ADEQUACY RATIO

• ADEQUACY RATIO
• (R) Actual Results
• 1 - (C) Results Without Program

Divided By
40
THE IMPACT STUDY REPORT INCLUDES

• Institutional enhancements as a result of
  implementing the program
• Enhancements to student and faculty performance,
  where applicable

41
THE IMPACT STUDY REPORT INCLUDES

• 3. Affect on administrative personnel
  effectiveness, where applicable
• 4. Overall academic and/or administrative
  program effectiveness enhancements
• Problems encountered that adversely affected
  the implementation of the program during the
  specified time- period

42
THE IMPACT STUDY REPORT INCLUDES

• Recommendations to strengthen the continued
  implementation of each program regardless of
  funding status

43
THE IMPACT STUDY REPORT INCLUDES

• Results of the Effectiveness Estimate analysis to
  determine the level of program effectiveness

44
THE IMPACT STUDY REPORT INCLUDES

• Results of the Adequacy Ratio analysis
  documenting the extent to which the program
  addressed the problem it was designed and
  implemented to address and the extent to which
  extramural funding had on program effectiveness.

45
CONCLUSION

• Information garnered through the impact study
  process, in combination with other sources of
  data, will provide an opportunity for the
  institutional leadership to incorporate the
  findings into programmatic and budgetary
  decision-making.

46
CONTACTS

• Dr. Gloria P. James gpryorjames_at_comcast.net
• Dr. Brenda R. Shaw bshaw_at_nccu.edu
• Mr. Samuel T. Rhoades strhoades_at_aol.com