Strategic Analysis/Evaluation Towards an assessment of change in research and innovation systems Dr. Stefan Kuhlmann, ISI

1
Strategic Analysis/EvaluationTowards an
assessment of change in research and innovation
systemsDr. Stefan Kuhlmann, ISI
speyer01.ppt
Folie 1
2
Evaluation may be defined as ...

• ... methodology-based analysis and assessment of
  the appropriateness of S/T policy assumptions and
  targets, of the related measures and their
  impacts, and of the goal attainment. (cf.
  Kuhlmann/Holland 1995a, 199 Kuhlmann/Meyer-Krahme
  r 1995, 3pp)

3
U r s a c h e u n d W i r k u n g. - Vor der
Wirkung glaubt man an andere Ursachen als nach
der Wirkung.Friedrich Nietzsche, Die fröhliche
Wissenschaft, Aphorismus 217, 1882
Cause and impact. - Before any impact you
believe in other causes than after the impact.
Folie 3
4
Main messages

• Think in terms of innovation systems

Folie 4
5
Innovation system limited reach of public policy
Source Technopolis 2000, modified and extended
by S. Kuhlmann, ISI
Folie 5
6
Main messages

• Think in terms of innovation systems
• Strategic evaluation combine performance
  assessments with policy analysis ( summative
  formative) analyse changing governance as a
  variable of policymaking!

Folie 6
7
Strategic paradigms of research and innovation
policies

• Market failure paradigm (Externalities
  Indivisibilities Risks)
• Mission paradigm, e.g. defence energy production
  and conservation, medicine and public health,
  space, and agriculture
• Co-operative policy paradigm, e.g. "Specific
  Programs" under the EU Framework Programs, or the
  "Verbundforschungsförderung" (co-operative RD
  between public sector institutes, universities,
  and industry) in Germany
• Need for structural change in the innovation
  system, e.g. government initiatives aiming at
  overcoming sclerotic institutions and procedures
  e.g. in the academic research system
• S/T for "public goods" as an aim of public
  investments (OECD 1995), e.g. sustainable
  development
• Counter-paradigm government failures, e.g.
  institutional inertia, lack of reliable
  information (on efficiency and impacts of
  policies etc.), lack of continuity and long-term
  perspective, red-tape procedures, rivalry of
  bureaucracies

8
Innovative research changing political
governance ...

• Tight public research budgets
• stronger performance orientation of progs. and
  institutions crucial
• orientation towards new public management in
  universities and research institutes
• Stronger political intervention aiming at
  modernisation for improved performance and
  innovation
• System evaluations (DFG/MPG FhG HGF) complain
  lack of inter-institutional cooperation, of
  interdisciplinarity, of international
  orientation, of support for junior researchers
• new laws pushing universities into (industrial)
  research markets (e.g. Baden-Württemberg)
• Federal government trying to redefine the
  missions of major institutions (e.g. FhG/GMD
  national research centres)
• Inflation of targeted funding of
  networks/clusters along the value chain
• Continuing EU integration (European Research
  Area)
• requires matching of EU-wide dispersed research
  facilities (centres of excellence)
• demand for common performance orientations

Folie 8
9
European multi-level system
Folie 9
10
Main messages

• Think in terms of innovation systems
• Strategic evaluation combine performance
  assessments with policy analysis ( summative
  formative) analyse changing governance as a
  variable of policymaking!
• Analysis for policy learning with respect to
  missions on different levels
• micro-level (lab-level researchers,
  institutes, sub-programs)
• meso-level (programs institutions)
• macro-level (research policymaking other public
  policies)

Folie 10
11
Strategic analysis (evaluation) of research
policies can draw on a broad range of methods
and indicators
Sectors, technologies,institutional
levels retrospectively, prospectively
Actors Companies Science Policymakers
Innovationprocesses micro, meso, macro
Folie 11
12
Main messages

• Think in terms of innovation systems
• Strategic evaluation combine performance
  assessments with policy analysis ( summative
  formative) analyse changing governance as a
  variable of policymaking!
• Evaluation for policy learning with respect to
  missions on different levels
• micro-level (lab-level researchers,
  institutes, sub-programs)
• meso-level (programs institutions)
• macro-level (research policymaking other public
  policies)
• Be aware of hybrid character of advanced
  research and innovation, cutting across
  public/private, disciplinary and basic/applied
  borderlines

Folie 12
13
Innovative research changed modalities of
knowledge production

• Mode 1 problems are set and solved in a context
  governed by the largely academic interests of a
  specific community. By contrast, Mode 2
  knowledge is carried out in a context of
  application. Mode 1 is disciplinary while Mode 2
  is transdisciplinary. Mode 1 is characterised by
  homogeneity, Mode 2 by heterogeneity.
  Organisationally, Mode 1 is hierarchical and
  tends to preserve its form, while Mode 2 is more
  heterarchical and transient (M. Gibbons et al.
  1994, 3).
• The emergence of a postmodern research system
  (A. Rip 1994)
• Changing performance dimensions (virtual lab).
• Increased share of contract research in
  universities and non-university research
  institutesfierce competition for contracts.
• Internationalisation of industrial research goes
  on.
• Increasing lack of highly qualified research
  staff, in public and private research.
• Increasing concerns/exepectations of various
  societal groups vis-a-vis research (e.g.
  genomics bio-tech ...).

Folie 13
14
Institutional borders blurred (virtual lab)-
(in advanced science and technology)

• Virtual laboratory network based polycentric
  knowledge production(e.g. neural nets plant
  biotechnology)
• networks are project-based
• with participants from different disciplines,
• heterogeneous institutions and culture behaviour
• Researchers have to cope with heterogeneous
  performance dimensions
• publication and teaching dominated by discipline
• targeted basic research characterised by
  interdisciplinary communication and interaction
• applied research led by industrial and economic
  rationales

15
Main messages

• Think in terms of innovation systems
• Strategic evaluation combine performance
  assessments with policy analysis ( summative
  formative) analyse changing governance as a
  variable of policymaking!
• Evaluation for policy learning with respect to
  missions on different levels
• micro-level (lab-level researchers,
  institutes, sub-programs)
• meso-level (programs institutions)
• macro-level (research policymaking other public
  policies)
• Be aware of hybrid character of advanced
  research and innovation, cutting across
  public/private, disciplinary and basic/applied
  borderlines
• Relate performance assessment to outcomes
  relevant to involved agencies, constituencies and
  societal stakeholders

Folie 15
16
Innovation policy stakeholders arena

• Differing interests, perspectives and values
• No dominant player?
• Contested policies
• Need for consensus?

Folie 16
17
Main messages

• Think in terms of innovation systems
• Strategic evaluation combine performance
  assessments with policy analysis ( summative
  formative) analyse changing governance as a
  variable of policymaking!
• Evaluation for policy learning with respect to
  missions on different levels
• micro-level (lab-level researchers,
  institutes, sub-programs)
• meso-level (programs institutions)
• macro-level (research policymaking other public
  policies)
• Be aware of hybrid character of advanced
  research and innovation, cutting across
  public/private, disciplinary and basic/applied
  borderlines
• Relate performance assessment to outcomes
  relevant to involved agencies, constituencies and
  societal stakeholders
• Create an infrastructure of Distributed
  Strategic Intelligence for research and
  innovation policymaking

Folie 17
18
Distributed Strategic Intelligence an
Architecture
Archit1.ppt
Folie 18
18
19
General principles of strategic intelligence

• Principle of participation strategic
  intelligence realises the multiplicity of actors
  and stakeholders values and interests involved
  in innovation policymaking (multiple perspective
  approach).
• Principle of "objectivisation" strategic
  intelligence "injects objectivised" information
  into the policy arena, i.e. the results of
  policy/strategy evaluations, foresight exercises
  or technology assessment, and also of analyses of
  changing innovation processes, of the dynamics of
  changing research systems and changing functions
  of public policies.
• Principle of mediation and alignment strategic
  intelligence facilitates debates and "discourses"
  between contesting actors in related policy
  arenas, thus mediating and "moderating",
  supported by "objectivised" information to be
  "digested" by the struggling parties.
• Principle of decision support strategic
  intelligence requires forums for negotiation and
  the preparation of policy decisions.

20
General requirements of distributed strategic
intelligence (DI)

• Networking requirement "infrastructures" for DI
  allow for multiple vertical and horizontal links
  amongst and across existing regional, national,
  sectoral, and transnational infrastructures and
  facilities of the related innovation systems and
  policy arenas.
• Active node requirement the infrastructure
  offers brokering "nodes" for managing the
  infrastructure. 3 types (a) enabling facilities,
  e.g. a "foresight bank". (b) "directory" allowing
  direct connections between relevant actors. (c)
  "register" for free access to all public
  strategic intelligence exercises undertaken.
• Transparent access requirement clear rules for
  access to DI, spanning from public domain
  information areas to restricted services charging
  a fee.
• Public support requirement to guarantee high
  degree of independence the DI infrastructure
  needs a regular and reliable support by public
  funding sources.
• Quality assurance requirement (a) bottom-up
  institutionalisation by providers of DI, e.g.
  professional associations (like AEA, EES,
  national ES). Scientific and expert journals
  university teaching (e.g. S/T policy programs"
  at US universities). (b) accreditation for DI
  providers, based on a vivid "scene" of experts.
  (c) reliable support with repeated and "fresh"
  strategic intelligence exercises and new
  combinations of actors, levels, and methods.

21

• Kuhlmann, S. / Boekholt, P. / Georghiou, L. /
  Guy, K. / Héraud, J.-A. / Laredo. Ph. / Lemola,
  T. / Loveridge, D. / Luukkonen, T. / Polt, W. /
  Rip, A. / Sanz-Menendez, L. / Smits, R. (1999)
  Improving Distributed Intelligence in Complex
  Innovation Systems. Brussels/Luxembourg (Office
  for Official Publications of the European
  Communities) (see also www.fhg.de/ti/final.pdf)