Logic Charts a Method for Program Planning and Evaluation: Applications to Research, Technology, Dev

1
Logic Charts - a Method for Program Planning and
Evaluation Applications to Research, Technology,
Development , and Deployment Policies and Programs

• Lecture organized by Platform Research and
  Technology Policy Evaluation on behalf of the
  Federal Ministry of Economics and Labour December
  9, 2008 Vienna, Austria
• Gretchen B. Jordan, Ph.D
• Sandia National Laboratories, U.S.A.
• gbjorda_at_sandia.gov

Some work presented here was completed for the
U.S. Department of Energy (DOE) Offices of
Science and Energy Efficiency and Renewable
Energy by Sandia National Laboratories,
Albuquerque, New Mexico, USA under Contract
DE-AC04-94AL8500. Sandia is operated by Sandia
Corporation, a subsidiary of Lockheed Martin
Corporation. Opinions expressed are solely those
of the author.
2
Presentation Outline

• Introduction to logic models
• RD policy logic model and evaluation
  considerations
• Example use in program description, performance
  measurement and evaluation
• More examples for research, for deployment
• Response to questions

2
3
LOGIC MODELING What and Why?

• The logic model concept was introduced in the
  1970s, has evolved to meet new needs, and is a
  basic tool for program management, evaluation and
  performance measurement.
• A logic model describes the theory and design of
  the program, how program activities and outputs
  influence program participants, customers and /
  or beneficiaries, leading to the achievement of
  the intended outcomes (short term, intermediate
  and long term).
• A logic model (diagram or table, with text) can
  describe a project, program, or portfolio of
  programs.
• A logic model provides the basis for
  accountability, by identifying key relationships
  and performance indicators linked to success
  along the results chain.

3
4
Definitions

• Resources Human and financial resources as well
  as other inputs required to support the program.
  Partnerships and alliances might be included
  here.
• Activities All those action steps necessary to
  produce program outputs includes delivery
  mechanisms.
• Outputs the products, goods, and services
  provided to direct customers.
• Outcomes Effects of activities and outputs
  resulting changes or benefits. Typically there
  are multiple, sequential outcomes. These are
  usually a restatement of program goals and
  objectives.
• External Influences Mediating factors that drive
  or restrain success

5
Logic Models Communicate About Program
Operations HOW the program will use resources
RD Policy or Program
Activities
Outputs
Resources

These elements are in the sphere of direct
control for policy and program decision makers
and implementers.
External Influences, context
6
Logic Models Communicate About WHO the Program
Targets and WHAT Happens Then
For/ With
RD Program or Policy
Results Chain
Customers/ Partners
Activities
Outputs
Short-Term Outcomes
Intermediate Outcomes
Long-Term Outcomes
Resources
(Includes Transfer/ Use)
Customer Decisions Actions
Strategic Goals
Strategic Objectives

These elements are in the sphere of direct
influence for RD policy or research
organizations.
External Influences, context
7
Logic Models Communicate About WHY the Program
Exists -- Goals
For/ With
Research Program
Results Chain
Customers/ Partners
Activities
Outputs
Short-Term Outcomes
Intermediate Outcomes
Long-Term Outcomes
Resources
(Includes Transfer, Use)
Customer Decisions Actions
Strategic Goals
Strategic Objectives

These elements are in the sphere of indirect
influence and include changes in socio-economic
factors.
External Influences, context
8
Multi-Year Planning During Logic Modeling Is
Then Tested and Measured During Implementation
Programs are designed from RIGHT to LEFT
PG performance goal APM annual performance
measure
Effective Transfer to Customers
Accountability Indicators
FY 05
Short-Term Outcomes
FY 04
Intermediate Outcomes
Long-Term Outcomes
PG 3
FY 03
Research Activity 1
Strategic Goals
Changes in Customer Knowledge, Decisions, Actions
Research Output 1
Strategic Objectives
PG 1
PG 2
Research Activity 2
Research Output 2
APM 1
Research Activity n
Research Output n
APM 2
Domain of Multi-Year Research Plans
External Influences, context
Programs are implemented managed from LEFT to
RIGHT
Adapted from Pahl Norland, March 2002
9
Steps logic model process

• Collect information through documents and perhaps
  establish a stakeholder workgroup.
• Define the problem and context for the program.
• 3. Define elements of the logic in a table.
• 4. Develop a diagram of logical relationships.
• Verify the program theory/logic with
  stakeholders, comparisons with implementation
  results
• Then use the logic model to develop or confirm
  performance measures for program monitoring and
  performance contracts, and in planning and
  evaluation.
• McLaughlin and Jordan, 1999, 2004

10
(No Transcript)
11
The hard part intermediate outcomes
Source Montague, www.pmn.net
Strategic Objectives e.g., wealth, health,
safety, environmental protection
STATE
Action/Adoption (Sustained Change)
BEHAVIORAL CHANGE in partners, stakeholders, and
target groups
Ability/ Capacity
Awareness/ Acceptance
Active Partner Support
The miracle in the middle
Legal/Business Climate
OPERATIONAL activities/outputs
Intensive Problem Solving/ RD
Technical Specialist Support
Information/ Advice
Awareness Building
Education
12
Sue Funnells Program Logic Matrix
Changes in attitudes of target businesses toward
being willing to change practices
Agreement to meet to discuss action Action
plans Specific examples of increased willingness
Promotes advisors and makes commitments about
confidentiality, etc.
business that request assistance, compared with
targets that do actions plans etc.
Admin. Records post program survey site
visits, etc.
Availability of confidential advisory assistance,
etc.
Business beliefs, past experiences, Etc.
Consumers purchase widget since she gets a rebate
Working through this matrix helps people to
specify outcomes and think through why the
program will or will not achieve each of these.
Consumers purchase the widget again without a
rebate
13
-we have defined what are right areas -we are
focused in those areas -turn over of portfolio

• Good planning
• -LDRD process
• -reward system reinforces

Right STE is available when needed
-Dynamic STE funding environment - stovepipes

• -strategic planning
• -LDRD management
• Rigorous peer review

(Later) OR Start from metrics work back)
(Later)
In areas that maintain, bring in new business,
each SMG has -trained staff -equipment -
discoveries
SMGs have differentiated capabilities
ID these capabilities Ability to attract
retain, partner Focused LDRD Education pipeline
Global labor markets Current shifts in
mission/funds in
Capabilities planning Allocate LDRD, mange LDRD
outcomes Fund students
(Later)
-targeted RD has good business case -involve
industry in planning - Document potential demand
-availability of capital -state/local
regulations -progress of alternative new
technologies -energy prices
Progress toward Commercial Launch of technology
EERE helped develop
-Industry investment in development -commercial sc
ale prototype produced
14
Many Possible LogicsMultiple arenas of research
technology development (RD) Multiple kinds of
innovations, intermediate ultimate outcomes
Innovation in Policy Procedures Products Proces
ses Knowledge Knowledge Tools
Economic, Social Mission Benefits
The idea innovation network Hage and
Hollingsworth (2000), modifying Kline and
Rosenberg (1986)
15
Linkage of Evaluation Issues to RD Policy
Evaluation

• Evaluation needs to consider the rationale for
  RD investments by government, that is,
  contribution to policy goals and the achievement
  of national economic, social and environmental
  objectives
• Government has three basic goals for funding ST
  / RD
• development of new knowledge, technical
  infrastructure, innovation capability and
  creation of highly qualified personnel
• Application of RD for increased national
  competitiveness, economic growth
• Application of RD for social and environmental
  well being, quality of life (public good)

12
16
Greg Tasseys model of the Innovation System
identifies different roles and contributions to
the economy.
Strategic
Production
Outputs/Market
Value-
Processes
Planning
Development
Added
Innovation
-
Economic Development
-
Public Health Safety
-
Environmental Protection
Risk
Reduction
Entrepreneurial Activity
Applied
Technologies

Generic
Technologies
Science Base
Economic, Social and Environmental Conditions

G. Tassey, National Institute of Standards and
Technology, U.S.A. 1991
13
17
Showing Government Policy Interventions in the
Innovation System
Production
Outputs/Market
Value-
Strategic
Planning
Processes
Development
Added
Innovation
-
Economic Development
-
Public Health Safety
-
Environmental Protection
Joint Industry
Funding and
ST Education and Promotion
Government Planning
Technical
Risk
Assistance
Reduction
Technology
National
Intellectual
Standards and
Research
Entrepreneurial Activity
Property
National Test
Facilities
Rights
Facilities
Applied
Technologies
Direct Funding
Generic
for universities,
Technologies
Federal labs, industry
Improve Research
and Production
Science Base
Efficiency
Economic, Social and Environmental Conditions
Derived from G. Tassey, National Institute of
Standards and Technology, U.S.A. 1991
14

18
Translating Policy Options Into a Simple RD
Logic Model What Area or Areas Require
Government Intervention? With What Mechanisms?
Sequence of Logical Outcomes
Concepts designs with possible
applications Knowledge spill-over
Investment by industry in innovative or advanced
commercial products
Favorable policies, capable delivery channels for
EERE products
Widespread adoption of EERE products More
productive use of energy
Potentially commercializ-able technologies to
replace existing or fill a system need
Program funding in appropriate areas Efficiency,
Fiscal responsibility
Relevant ST expertise, capabilities and
facilities to deliver programs
Feedback Loops (not a linear process)
EERE Energy Efficiency and Renewable Energy
Policy Options (Tassey Model)
ST Education and Promotion
19
Linking Logic Models to Program Monitoring and
Evaluation An Example

• The U.S. Department of Energys Office of Energy
  Efficiency and Renewable Energy (EERE) includes
  programs from research to utilization.
• A logic model of EEREs portfolio of linked
  programs was developed.
• This can be used by others as a generic RD
  logic model.
• EEREs goals are to
• Modernize energy conservation
• Increase energy supplies
• Modernize our critical energy infrastructure

20
Office of Energy Efficiency and Renewable Energy
(EERE)

• EERE accomplishes its mission through 10
  Technology Development (TD) Programs and the
  Office of Technology Advancement and Outreach
  (TAO)
• The 10 TD Programs are
• Fuels Vehicles
• Vehicles Technologies
• Biomass/Biofuels
• Hydrogen
• Power Generation
• Wind
• Solar
• Geothermal
• Energy Efficiency
• Building Technologies
• Industrial Technologies
• Weatherization
• Federal Energy Management
• These research, development, demonstration, and
  deployment (RD3) programs invest in high-risk,
  high-value RD that accelerates the development
  of advanced clean energy technologies and
  practices. These programs also have deployment
  components and activities that address market and
  behavioral barriers to specific technologies that
  facilitate the deployment of advanced
  technologies and practices that may be either
  currently available or in the RD pipeline for
  future deployment.

21
EERE has 7 different strategies and multiple
policy instruments. The strategies are
represented as activities in the logic model
Inputs
Program planning assessment
Conduct research
Develop technology
Demonstrate technology
Deploy technology
Develop maintain program infrastructure
Developing government market infrastructure
Activities
Feedback Loops
Outputs
For/ With

• Note the logic in a complex model is both
• left to right
• top to bottom.

Outcomes
22
Outputs and a sequence of outcomes for each
activity are in the columns
Inputs
Demonstrate technology
Activities
Test, improve, validate commercial- scale
technology, Give industry hands-on experience
Outputs Outcomes directly influenced
Relevant industries
For/ With
Investment by industry in innovative or advanced
commercial products
Outcomes
23
EEREs draft logic model shows how its
strategies/activities are linked to its goals
External Influences
Federal, state local government funding Private
funding, Personnel, Facilities, Past RD results
Political environment Quality of RD
proposals Unpredictable nature of RD Cost and
performance of competing technologies Industry
willingness to take risk Energy prices State of
the economy Government policies and regulations
Inputs
Program planning assessment
Conduct research
Develop technology
Demonstrate technology
Deploy technology
Develop maintain program infrastructure
Developing government market infrastructure
Activities
Feedback Loops
Benefit estimates, Priorities identified, Budget
requests, Program plans
New knowledge, proof of concepts as represented
by data, publications
Technology prototypes -initial -intermediate
-commercial Performance analysis
Test, improve, validate commercial- scale
technology, Give industry hands-on experience
Government purchases, Information
disseminated, Early seeding of technologies
Public private labs and test beds, Knowledge
bases, Trained ST personnel, Partnerships
Codes and standards, Trained personnel, Audits
tools, State programs
Outputs
Programs, CFO, OMB, Congress
RD Community, Industry
Programs, partners
RD community
Relevant industries
Relevant markets
Potential purchasers
For
Concepts designs with possible
applications, Knowledge spill-over
Investment by industry in innovative or advanced
commercial products
Favorable policies, capable delivery channels for
EERE products
Widespread adoption of EERE products More
productive use of energy
Potentially commercializ-able technologies to
replace existing or fill a system need
Program funding in appropriate areas Efficiency,
Fiscal responsibility
Relevant ST expertise, capabilities and
facilities to deliver programs
Outcomes
Economic, security, and environmental
benefits Technology leadership
Spin-off products and their associated
benefits New products businesses
National RD capabilities, including options if
circumstances change
24
Each box in the logic model is a potential
measurement area
25
Performance targets may also be developed for
each box in the logic model
26
Arrows between the boxes help identify evaluation
questions
Evaluation Questions
Demonstrate technology
Activities

• Has EERE engaged the right partners in tests and
  done it efficiently?

Test, improve, validate commercial- scale
technology, Give industry hands-on experience

• Have relevant industries gained hands-on
  experience with the technologies?

Outputs

• Has industry experience lead to investment in
  these innovative or advanced commercial energy
  products?

Relevant industries
For With
Investment by industry in innovative or advanced
commercial products

• Have EEREs efforts to test and demonstrate
  energy technologies led to validated commercial
  scale technologies? (Activities to outcomes)

Outcomes
27
Types of evaluation examine different aspects of
performance
Outcome or Impact Evaluation What are the
outcomes associated with the program? Intended
and unintended? Did the program cause the
outcomes?
Intermediate Outcomes (through customers)
Long-Term Outcomes Problem Solution
Resources (Inputs)
Activities
Outputs
Short-Term Outcomes
for Customers Reached
Process Evaluation Is the program being
implemented as planned?
External Influences and Related Programs
(mediating factors)
Market Assessment Is there a need? Has context
changed?
28
Evaluations can explain why some goals were met
and others were not
(3) But, competing technologies improved more
than expected
(1) Funding was about what was expected
(4) Energy prices were lower than expected
(2) RD yielded expected results
(5) Thus, market penetration was slower than
expected
(6) And benefits were lower than expected
29
In summary, logic models for RD programs help
identify
30
Selected References
The Logic Model Guidebook Better Strategies for
Great Results, Cynthia Phillips, Lisa Wyatt
Knowlton, Editors, September 2008 Sage. Reed,
John H, G. Jordan, Using Systems Theory and Logic
Models to Define Integrated Outcomes and
Performance Measures in Multi-program Settings,
in Research Evaluation, Volume 16 Number 3
September 2007. Funnell, S. (2000). Developing
and Using a Program Theory Matrix for Program
Evaluation and Performance Monitoring, in New
Directions for Evaluation, Rogers, et.al. Eds.,
San Francisco Jossey-Bass, Number 87, Fall, pp.
91-102. McLaughlin, John A., and Jordan,
Gretchen B., Chapter 1 Logic Models, in
Handbook of Practical Program Evaluation, 2nd
Edition, Wholey, J., Hatry, H., and Newcomer, K.,
Eds., Jossey-Bass, 2004. McLaughlin, John A.,
and Jordan, Gretchen B., Logic Models A Tool
for Telling Your Performance Story, Evaluation
and Program Planning, Elsevier Science New York,
Vol. 22, Issue 1, February 1999, Pp.
65-72. University of Wisconsin Cooperative
Extension (2003). Evaluation Logic Model
Bibliography, Program Development and Evaluation
web site, http//www.uwex.edu/ces/pdande/evaluatio
n/evallogicbiblio.html. See also www.
WREN-network.net
30
G. Jordan 2008
31
More Examples of Logic Modelsfor Research, for
Deployment--with emphasis on Outcomes --
32
Measurement framework includes 5 inter-related
areas that will provide useful data
External Factors
Program Management
Technology Readiness
External Factors
Market Readiness
Ultimate Outcomes
(Feedback loops not shown)
33
Generic measures for EERE programs
Draft 08/03/07
Technology Attractiveness Market Acceptance
34
Logic Flow of the EERE Solar Program
Draft 09/11/07
Technology Readiness Projects
Market Readiness Projects
RD on Disruptive Technologies - CSP
Towers - Solar hybrid lighting
Applied RD - Materials devices - 3rd gen.
PV - Advanced CPV - Solar hydrogen
System Development - 1st 2nd Generation PV -
Silicon PV - CSP trough

• Testing
• Evaluation
• - Techniques
• Facilities
• Validation

• Business Support
• -RDD on
• -- Manufacturing
• -Built in PV
• training

Policy Knowledge Tech support for codes,
policy, knowledge base
End User Assistance -Tech support -demos -outreach
Activities
External Factors
RD advances (non-stage gate)
Assured performance and compatibility Lower risk
End users aware Integrate into facilities design
Tech. scale up, lower costs Improved
design Certified installers
Studies disseminated Model legislation Robust
info channels States adopt best practice

• Improvement in component/system
• - Efficiency - Reliability - Lifetime
• Capital cost - OM cost

Outputs

• Components/systems moves thru stages
• Preliminary investigation
• Detailed investigation
• Development
• Validation
• Commercial launch

Supply chain is in place and profitable
Supportive codes, policies, and public entities
End users persuaded to purchase technology
EERE knowledge transferred utilized in further
RD or unrelated products
Economically attractive technology available
Options value of non-commercialized technologies
Market penetration of technology - Early
adoption - Replication - Growing demand -
Wide-spread use
Outcomes
Solar lighting in homes businesses
Residential electricity generation (PV)
Commercial electricity generation (PV)
Utility-scale electricity generation (CSP, CPV)
Fuel diversity, oil savings, load
reduction, energy system cost savings, emission
reductions, U.S. jobs
35
The Logic of a Basic Research Project
Manage Resources expenditures by types of
activities, skilled staff, core competencies
environment for quality research, soundness of
research planning and evaluation, use scientific
method
Exchange knowledge in papers, conferences, etc.
Identify and state the problem
Do research and report findings
Develop, test and build research tools
Reach targeted partners and customers other
researchers, laboratories, students,
universities, applied researchers and technology
developers, industry attendees at conferences,
readers of publications
Activities
Growing consensus on problems
New techniques to research problems
Growing convergence on solutions to problems
Apply ideas of others in research
Outcomes and Results
New disciplines New insights and knowledge
Feedback loops are not shown
Potential impacts of research
Use in RD or Commercialization
Actual impacts of the research
G. Jordan 1996
36
Logic Model of a Program of Basic Research (U.S.
DOE DRAFT -Unofficial)
ACTIVITIES
Identify/ Direct/Redirect resources to important
questions needs
Gather/ Build/ Maintain/ Provide resources in
select areas
Disseminate/ Seek Review/ Feedback research
plans, findings
Perform or Have Performed high quality research
Students work with DOE or elsewhere
Facility use - DOE others
Construct, operate, facilities
Propose Experiment, theorize Collect analyze
data
OUTPUTS OUTCOMES
Inform and be informed by collaborators, peers,
potential users
Prove, disprove Theories, techniques developed
solutions generated
Strong communities of practice
Robust ST workforce
Robust ST Facilities Equipment
New structure, new ideas, tools, fields,
Opportunities for use by others
Transitions findings used
Capacity/Agility
Significant Contributions to DOE Mission,
National Needs, Society
G. .Jordan 05/13/2002
37
A Framework for organizing the scientific
questions and research topics needed to create
the scientific foundation for environmental
decisions - Particulate Matter Research
SOURCES / EMISSIONS
HEALTH DISEASE RISK

• Respiratory
• Cardiovascular

• Chemical
• Physical
• Microbial

TRANSPORT / TRANSFORMATION
ALTERED STRUCTURE / FUNCTION

• Kinetics
• Thermodynamics
• Chemistry
• Dispersion
• Meteorology

• Edema
• Arrythemia
• Asthma

EARLY BIOLOGICAL EFFECT
AMBIENT ENVIRONMENTAL CONDITIONS
ACTIVITY PATTERNS

• Molecular
• Biochemical
• Cellular
• Organ
• Organism

Susceptible Individual Susceptible
Subpopulation Population Distribution
DOSE

• Air
• Water

• Absorbed
• Internal
• Target
• Biological Effective

HUMAN OR ECOSYSTEM EXPOSURE

• Individual
• Community
• Population

Change in ambient environmental concentrations
Change in magnitude, duration, frequency of
exposure
Change in emissions or discharge
Early change in function, structure, or effect
Change in human or ecosystem health
Change in uptake and / or assimilation
Research reduces uncertainty across the
health-to-source paradigm and in critical links
related to sources, exposure, health effects,
risk assessment, and regulatory decision-making
Source US EPA, Pahl, et al, AEA 2007
38
Source US DOE, Reed Jordan
39

• We need to describe and measure the expected
  response of
• Knowledge workers
• Public Entities
• Businesses and manufacturing
• End-users

Knowing these activities and their corresponding
outputs
To show how activities are connected to impacts
Source US DOE, Reed Jordan
40
Diffusion of Innovations a model(applies in
each domain)
41
A detailed deployment logic model
42
Thank you.

• Questions and Answers