Title: Strategic Analysis/Evaluation Towards an assessment of change in research and innovation systems Dr. Stefan Kuhlmann, ISI
1Strategic Analysis/EvaluationTowards an
assessment of change in research and innovation
systemsDr. Stefan Kuhlmann, ISI
speyer01.ppt
Folie 1
2Evaluation may be defined as ...
- ... methodology-based analysis and assessment of
the appropriateness of S/T policy assumptions and
targets, of the related measures and their
impacts, and of the goal attainment. (cf.
Kuhlmann/Holland 1995a, 199 Kuhlmann/Meyer-Krahme
r 1995, 3pp)
3U r s a c h e u n d W i r k u n g. - Vor der
Wirkung glaubt man an andere Ursachen als nach
der Wirkung.Friedrich Nietzsche, Die fröhliche
Wissenschaft, Aphorismus 217, 1882
Cause and impact. - Before any impact you
believe in other causes than after the impact.
Folie 3
4Main messages
- Think in terms of innovation systems
Folie 4
5Innovation system limited reach of public policy
Source Technopolis 2000, modified and extended
by S. Kuhlmann, ISI
Folie 5
6Main messages
- Think in terms of innovation systems
- Strategic evaluation combine performance
assessments with policy analysis ( summative
formative) analyse changing governance as a
variable of policymaking!
Folie 6
7Strategic paradigms of research and innovation
policies
- Market failure paradigm (Externalities
Indivisibilities Risks) - Mission paradigm, e.g. defence energy production
and conservation, medicine and public health,
space, and agriculture - Co-operative policy paradigm, e.g. "Specific
Programs" under the EU Framework Programs, or the
"Verbundforschungsförderung" (co-operative RD
between public sector institutes, universities,
and industry) in Germany - Need for structural change in the innovation
system, e.g. government initiatives aiming at
overcoming sclerotic institutions and procedures
e.g. in the academic research system - S/T for "public goods" as an aim of public
investments (OECD 1995), e.g. sustainable
development - Counter-paradigm government failures, e.g.
institutional inertia, lack of reliable
information (on efficiency and impacts of
policies etc.), lack of continuity and long-term
perspective, red-tape procedures, rivalry of
bureaucracies
8Innovative research changing political
governance ...
- Tight public research budgets
- stronger performance orientation of progs. and
institutions crucial - orientation towards new public management in
universities and research institutes - Stronger political intervention aiming at
modernisation for improved performance and
innovation - System evaluations (DFG/MPG FhG HGF) complain
lack of inter-institutional cooperation, of
interdisciplinarity, of international
orientation, of support for junior researchers - new laws pushing universities into (industrial)
research markets (e.g. Baden-Württemberg) - Federal government trying to redefine the
missions of major institutions (e.g. FhG/GMD
national research centres) - Inflation of targeted funding of
networks/clusters along the value chain - Continuing EU integration (European Research
Area) - requires matching of EU-wide dispersed research
facilities (centres of excellence) - demand for common performance orientations
Folie 8
9European multi-level system
Folie 9
10Main messages
- Think in terms of innovation systems
- Strategic evaluation combine performance
assessments with policy analysis ( summative
formative) analyse changing governance as a
variable of policymaking! - Analysis for policy learning with respect to
missions on different levels - micro-level (lab-level researchers,
institutes, sub-programs) - meso-level (programs institutions)
- macro-level (research policymaking other public
policies)
Folie 10
11 Strategic analysis (evaluation) of research
policies can draw on a broad range of methods
and indicators
Sectors, technologies,institutional
levels retrospectively, prospectively
Actors Companies Science Policymakers
Innovationprocesses micro, meso, macro
Folie 11
12Main messages
- Think in terms of innovation systems
- Strategic evaluation combine performance
assessments with policy analysis ( summative
formative) analyse changing governance as a
variable of policymaking! - Evaluation for policy learning with respect to
missions on different levels - micro-level (lab-level researchers,
institutes, sub-programs) - meso-level (programs institutions)
- macro-level (research policymaking other public
policies) - Be aware of hybrid character of advanced
research and innovation, cutting across
public/private, disciplinary and basic/applied
borderlines
Folie 12
13Innovative research changed modalities of
knowledge production
- Mode 1 problems are set and solved in a context
governed by the largely academic interests of a
specific community. By contrast, Mode 2
knowledge is carried out in a context of
application. Mode 1 is disciplinary while Mode 2
is transdisciplinary. Mode 1 is characterised by
homogeneity, Mode 2 by heterogeneity.
Organisationally, Mode 1 is hierarchical and
tends to preserve its form, while Mode 2 is more
heterarchical and transient (M. Gibbons et al.
1994, 3). - The emergence of a postmodern research system
(A. Rip 1994) - Changing performance dimensions (virtual lab).
- Increased share of contract research in
universities and non-university research
institutesfierce competition for contracts. - Internationalisation of industrial research goes
on. - Increasing lack of highly qualified research
staff, in public and private research. - Increasing concerns/exepectations of various
societal groups vis-a-vis research (e.g.
genomics bio-tech ...).
Folie 13
14Institutional borders blurred (virtual lab)-
(in advanced science and technology)
- Virtual laboratory network based polycentric
knowledge production(e.g. neural nets plant
biotechnology) - networks are project-based
- with participants from different disciplines,
- heterogeneous institutions and culture behaviour
- Researchers have to cope with heterogeneous
performance dimensions - publication and teaching dominated by discipline
- targeted basic research characterised by
interdisciplinary communication and interaction - applied research led by industrial and economic
rationales
15Main messages
- Think in terms of innovation systems
- Strategic evaluation combine performance
assessments with policy analysis ( summative
formative) analyse changing governance as a
variable of policymaking! - Evaluation for policy learning with respect to
missions on different levels - micro-level (lab-level researchers,
institutes, sub-programs) - meso-level (programs institutions)
- macro-level (research policymaking other public
policies) - Be aware of hybrid character of advanced
research and innovation, cutting across
public/private, disciplinary and basic/applied
borderlines - Relate performance assessment to outcomes
relevant to involved agencies, constituencies and
societal stakeholders
Folie 15
16 Innovation policy stakeholders arena
- Differing interests, perspectives and values
- No dominant player?
- Contested policies
- Need for consensus?
Folie 16
17Main messages
- Think in terms of innovation systems
- Strategic evaluation combine performance
assessments with policy analysis ( summative
formative) analyse changing governance as a
variable of policymaking! - Evaluation for policy learning with respect to
missions on different levels - micro-level (lab-level researchers,
institutes, sub-programs) - meso-level (programs institutions)
- macro-level (research policymaking other public
policies) - Be aware of hybrid character of advanced
research and innovation, cutting across
public/private, disciplinary and basic/applied
borderlines - Relate performance assessment to outcomes
relevant to involved agencies, constituencies and
societal stakeholders - Create an infrastructure of Distributed
Strategic Intelligence for research and
innovation policymaking
Folie 17
18 Distributed Strategic Intelligence an
Architecture
Archit1.ppt
Folie 18
18
19General principles of strategic intelligence
- Principle of participation strategic
intelligence realises the multiplicity of actors
and stakeholders values and interests involved
in innovation policymaking (multiple perspective
approach). - Principle of "objectivisation" strategic
intelligence "injects objectivised" information
into the policy arena, i.e. the results of
policy/strategy evaluations, foresight exercises
or technology assessment, and also of analyses of
changing innovation processes, of the dynamics of
changing research systems and changing functions
of public policies. - Principle of mediation and alignment strategic
intelligence facilitates debates and "discourses"
between contesting actors in related policy
arenas, thus mediating and "moderating",
supported by "objectivised" information to be
"digested" by the struggling parties. - Principle of decision support strategic
intelligence requires forums for negotiation and
the preparation of policy decisions.
20General requirements of distributed strategic
intelligence (DI)
- Networking requirement "infrastructures" for DI
allow for multiple vertical and horizontal links
amongst and across existing regional, national,
sectoral, and transnational infrastructures and
facilities of the related innovation systems and
policy arenas. - Active node requirement the infrastructure
offers brokering "nodes" for managing the
infrastructure. 3 types (a) enabling facilities,
e.g. a "foresight bank". (b) "directory" allowing
direct connections between relevant actors. (c)
"register" for free access to all public
strategic intelligence exercises undertaken. - Transparent access requirement clear rules for
access to DI, spanning from public domain
information areas to restricted services charging
a fee. - Public support requirement to guarantee high
degree of independence the DI infrastructure
needs a regular and reliable support by public
funding sources. - Quality assurance requirement (a) bottom-up
institutionalisation by providers of DI, e.g.
professional associations (like AEA, EES,
national ES). Scientific and expert journals
university teaching (e.g. S/T policy programs"
at US universities). (b) accreditation for DI
providers, based on a vivid "scene" of experts.
(c) reliable support with repeated and "fresh"
strategic intelligence exercises and new
combinations of actors, levels, and methods.
21- Kuhlmann, S. / Boekholt, P. / Georghiou, L. /
Guy, K. / Héraud, J.-A. / Laredo. Ph. / Lemola,
T. / Loveridge, D. / Luukkonen, T. / Polt, W. /
Rip, A. / Sanz-Menendez, L. / Smits, R. (1999)
Improving Distributed Intelligence in Complex
Innovation Systems. Brussels/Luxembourg (Office
for Official Publications of the European
Communities) (see also www.fhg.de/ti/final.pdf)