Some Thoughts About Evaluating nsfs investments in informal science education

1
Some Thoughts About Evaluating nsfs investments
in informal science education

• Dr. Mark St. John
• Inverness Research Associates
• January 2005
• www.inverness-research.org

2
INVERNESS RESEARCH
3
Inverness California
4
Inverness California
5
The Functions of Evaluation
6
Functions of Evaluation

• Conceptualization, Documentation And Portrayal
• Formative Feedback
• Planning and Opportunity Analysis
• Summative Evaluation
• Research

7
Functions of Evaluation
8
EVALUATION AS A PROCESS OF GROUNDTRUTHING
9
EVALUATING NSF INVESTMENTS

• NSF

BENEFITS TO EDUCATION
PROJECT OR ACTIVITY
UNDERLYING THEORY OF ACTION
10
Groundtruthing involves the comparison of theory
of action (mental models) and field realities
11
Evaluation as Groundtruthing Comparing Project
Logic and Field Realities
Project Theory of Action (Assumptions,
Intentions, Strategies Design Principles)
Field Realities
Evaluation helps to elucidate and refine
Evaluation determines what is actually happening
Evaluation Focuses On Congruence
12
1) Conceptualization, Documentation and Portrayal

• Foundational function
• Focuses on both theory of action and actual work
  done
• Assists the project in basic documentation
• Lays out the project in complete and analytical
  fashion so that all can see what it intends to be
  and what it is actually doing
• Helps others understand the nature of the work of
  the project

13
2) Formative Feedback

• Critical friend role Groundtruthing
• Working in relationship
• Focus on issues, interactions, and information
  that will assist project in shaping design and
  implementation
• Supports the co-evolution of project and
  evaluation design

14
3) Planning and opportunity analysis
THE STRENGTHS AND ASSETS OF THE PROJECT
IMPORTANT EDUCATIONAL NEEDS
THE CURRENT LANDSCAPE OF SOCIETY (TRENDSECONOMIC
S, POLITICS, CULTURE)
15
3) Summative Evaluation

• Helps others understand the theory, work and
  contributions of the project
• Allow funders (and others) to assess return on
  investment and overall value of the effort
• Places the project in broader contexts
• Seeks creative methods to measure appropriate
  outcomes in rigorous ways
• Can support the case for future funding

16
SUMMATIVE MAKING THE CASE (TYPES OF CLAIMS)
CLAIMS THAT ARE TRUE
CLAIMS THAT HAVE POLITICAL CURRENCY
CLAIMS THAT ARE DOCUMENTABLE
17
SUMMATIVE MAKING A CASE DIFFERENT DIMENSIONS
OF THE ARGUMENT AND DIFFERENT TYPES OF EVIDENCE
18
4) Research

• Treats the project as a case
• Helps the project summarize lessons learned
• Studies the landscape the project works within
• Studies the more general lessons learned
  vis-à-vis this type of investment
• Helps the project generate knowledge as one of
  its contributions

19
Suggested Steps in Project Evaluation

• Prioritize (5) functions
• Begin work with your evaluator around the theory
  of action
• Use groundtruthing in a formative fashion
• Decide on summative audience and their needs
• Make the case using 1) project logic and 2)
  initiative goals or drivers
• Decide if and how to use the evaluation to
  contribute to research effort

20
Evaluating An investment in informal science
education is evaluating a particular kind of NSF
investment
21
Different perspectives on informal science
education
22
I, Thou, IT the relationship triangle
The Watershed And its Phenomena (physical,
biological, historical, cultural Economic)

• An Evironment
• and Institutional
• Context

The Teacher, Guide, Adults Exhibit Designer
The Vistor, Participant, Student
23
Organizational health
Mission
Personal and Collective Energy
Structures, Relationships
Resources
24
Informal science learningCULTURES AND
SUBCULTURES
MAINSTREAM CULTURE
SCIENCE INTERESTED SUBCULTURE
ENVIRONMENT INTERESTED SUBCULTURE
25
THE SPECTRUM OF ACCULTURATION
DIS-INTERESTED
HIGHLY INTERESTED
INFORMAL EXPERIENCES
FORMAL EXPERIENCES
26
AUDIENCES AND TEACHERS
PROFESSIONALS
ADVANCED AMATEURS
ATTENTIVE PUBLIC
READY PUBLIC
MAINSTREAM PUBLIC
27
NSFs Relationship to the Informal Learner

• (NSF To Learner
• 3 Degrees Of Separation)

28
The First Degree
SCIENCE LEARNING AND ACCULTURATION
INFORMAL LEARNING OPPORTUNITIES
SEMI-FORMAL LEARNING OPPORTUNITIES
FORMAL LEARNING OPPORTUNITIES
29
The SECOND Degree
SCIENCE LEARNING AND ACCULTURATION
INFORMAL LEARNING OPPORTUNITIES
FORMAL LEARNING OPPORTUNITIES
SEMI-FORMAL LEARNING OPPORTUNITIES
INSTITUTIONS AND SYSTEMS (Formal, Semi-Formal,
Informal)
30
The THIRD Degree
PUBLIC LEARNING AND ACCULTURATION
INFORMAL LEARNING OPPORTUNITIES
FORMAL LEARNING OPPORTUNITIES
SEMI-FORMAL LEARNING OPPORTUNITIES
INSTITUTIONS AND SYSTEMS (Formal, Semi-Formal,
Informal)
PROJECTS
ADD VALUE
NSF
31
Simultaneously funding the capacity and the work
of the project
Develop capacity to do the work
INVESTMENT
Project activities to do the work