Economia%20delle%20ICT

1
Moscow January 21, 2015 An overview of STI
statistics and indicators Giorgio
Sirilli ISCrES - CNR
2
Outline of the presentation

• STI indicators
• STI policy
• The actors
• Models of innovation
• Manuals
• Uses and abuses of indicators
• Concluding remarks

3
Science, technology and innovation

• Science is the reasoned investigation or study of
  phenomena, aimed at discovering enduring
  principles among elements of the phenomenal world
  by employing formal techniques such as the
  scientific method
• Technology is the collection of tools, including
  machinery, modifications, arrangements and
  procedures used by humans
• Innovation is the process practical application
  of knowledge translating an idea or invention
  into a good or service that creates value or for
  which customers will pay

4
Indicators

• Statistic A numerical fact or datum, i.e. one
  computed from a sample.
• Statistical data Data from a survey or
  administrative source used to produce statistics.
• Statistical indicator A statistic, or
  combinations of statistics, providing information
  on some aspect of the state of a system or of its
  change over time. (For example, gross domestic
  product (GDP) provides information on the level
  of value added in the economy, and its change
  overtime is an indicator of the economic state of
  the nation.)

5
Indicators

• Indicators are a technology, a product, which
• - governs behaviour
• - is modified by users (outside of the producer
  community)
• - develops in response to user needs
• Data sources
• Surveys, administrative data, private files, case
  studies
• Data collection is informed by manuals
• Data populate statistics which can be indicators
• Decisions are taken on the basis of indicators

6
Science and technology indicators

• ST indicators are defined as a series of data
  which measures and reflects the science and
  technology endeavor of a country, demonstrates
  its strengths and weaknesses and follows its
  changing character notably with the aim of
  providing early warning of events and trends
  which might impair its capability to meet the
  countrys needs.
• Indicators can help to shape lines of argument
• and policy reasoning. They can serve as
• checks, they are only part of what is needed.
• (OECD, 1976)

7
Lord Kelvin

• If you can not measure it, you can not improve
  it.
• Lord Kelvin

8
Hariolf Grupp

• If you can not measure it, dont talk about it.
  
• Ariolf Grupp

9
The evolution of STI indicators

• The first attempt to measure ST in 1957
• Frascati Manual (1963)
• The Frascati manual family
• A continous process of broadening and deepening
  from macro to micro, from public to private
• The role of international organisations
• The dialogue between producers and users

10
Science and technology policy
A history which starts after WW2
1940s
2010s
competitiveness
social needs
innovation
science
technology
jobs
11
Science and technology policy

• Report Science the Endless Frontier 1945
  (Vannevar Bush)

12
Science the Endless Frontier

• Concerns Military security Health
• Solution Science policy
• The Government is particularly fitted to perform
  certain functions, such as the coordination and
  support of broad programs on problems of great
  national importance
• Scientific progress on a broad front results
  from the play of free intellects, working on
  subjects of their own choice, in the manner
  dictated by their curiosity for exploration of
  the unknown. Freedom of inquiry must be preserved
  under any plan for Government support of science

13
Science the Endless Frontier

• Publicly and privately supported colleges and
  universities and the endowed research institutes
  must furnish both the new scientific knowledge
  and the trained research workers. It is chiefly
  in these institutions that scientists may work in
  an atmosphere which is relatively free from the
  adverse pressure of convention, prejudice, or
  commercial necessity. At their best they provide
  the scientific worker with a strong sense of
  solidarity and security, as well as a substantial
  degree of personal intellectual freedom.
• Industry is generally inhibited by preconceived
  goals, by its own clearly defined standards, and
  by the constant pressure of commercial necessity.
  Satisfactory progress in basic science seldom
  occurs under conditions prevailing in the normal
  industry laboratory.

14
The triangle in STI policy making

• Policy makers design the future
• Analysts interpret today
• Data producers measure the past

15
Luigi EinaudiItalian President
Policy makers
Conoscere per deliberare Know first and then
sanction
16
Policy makers

• Vannevar Bush
• Report Science the Endless Frontier 1945

• President Roosevelt

17
Policy makers

• Why shoud we pay the
• researchers if we make the
• best shoes in the world?
• Silvio Berlusconi

Culture does not provide food (Con la cultura
non si mangia) Giulio Tremonti
18
The relationship between producers and users

• After WWII the main issue in STI policy was the
  social responsibility of science. Now the
  attention is placed on the social return of
  investment on STI and, in particular, on
  innovation and social objectives
• Weak innovation theories do not allow a
  straighforward interpretation of indicators
• A tension the simplification of policy makers
  (e.g. the 3 RD/GDP ratio) versus deepening of
  analysts
• The time dimension demand of indicators and
  analyses (quick and dirty) but . the
  construction of indicators is a heavy ship
  (more than 10 years to build)

19

Producers of indicators

• OECD Group of National Experts on Science and
  Technology Indicators - NESTI

20
NESTI

• NESTI Working Party of National Experts on
  Science and Technology Indicators (1962)
• Mission
• Produce methodologies, statistics and analyses
• Co-ordinating body
• Clearing house

21
International organisations active in STI
indicators

• OECD and member countries
• Eurostat and the relevant Commission DGs and
  member states of the EU
• UNESCO
• United Nations Institute of Statistics
• RICYT and member countries
• NEPAD is moving in this direction

22
The role of OECD in the development of ST
measurement in the 1960s

• In the 1960s few countries collected data on ST
• The OECD offered a ready-made model for those who
  had not yet developed the necessary instruments
• Standardisation was proposed by an international
  organisation and not by a specific country
• The Frascati Manual was introduced with a petits
  pas strategy
• - the first edition was an internal document
  only (1962)
• - the Manual was tested (1963-1964) in many
  countries
• - it was revised in light of the experience
  gained from the surveys
• The Frascati Manual has never been an imperative
  document countries are totally free to apply its
  conventions
• No harmonised questionnaire

23
United Nations Fundamental Principles of Official
Statistics

• Official statistics provide an indispensable
  element in the information system of a democratic
  society. They need to be compiled and made
  available on an impartial basis by official
  agencies.
• To retain trust in official statistics, the
  statistical agencies need to decide according to
  strictly professional considerations on methods
  and procedures.
• To facilitate a correct interpretation of the
  data, the statistical agencies are to present
  information according to scientific standards on
  the sources, methods and procedures of the
  statistics.

Source UN Statistical commission
24
United Nations Fundamental Principles of Official
Statistics

• Data for statistical purposes may be drawn from
  all types of sources, be they statistical surveys
  or administrative records.
• Individual data collected by statistical agencies
  are to be strictly confidential and used
  exclusively for statistical purposes.
• The laws, regulations and measures under which
  the statistical systems operate are to be made
  public.
• Coordination among statistical agencies is
  essential.

Source UN Statistical commission
25
The dangerous business of statisticians

• What matters is not how one fashions things, but
  what one does with them not the weapon, but the
  battle (). The making and the using of the tool
  are different things.
• O. Spenger, Man and Technics A Contribution to
  a Philosophy of Life, 1932

26
Indicators community

• The members of the indicators community have to
  play the difficult game of finding a proper
  balance between data collection, analysis and
  policy making preserving their intellectual
  integrity

27
Models of innovation

• Indicators make sense in the context of models

Linear Chain-linked Triple helix Open innovation
28
The Linear model of innovation
Research
Production
Development
Design
Engineering

• Based on research
• Sequential
• Technocratic

29
The Chain-linked model of innovation
Research

• Ricerca

Knowledge
Invention/ analytical project
Detailed project test
Re-design production
Distribution market
Potential market

• Invenzione/
• progettazione
• analitica

• Progettazione
• dettagliata e
• test

• Distribuzione
• e mercato

• Mercato
• potenziale

Based on design Interactive Research is not a
pre-requisite for innovation
30
The Triple-helix model of innovation
31
The Triple-helix model of innovation
Government Government-sponsored Research
Institutes Research for public purposes Applied
research Universities Supply scientists
engineers Curiosity-driven Basic
research Firms Commercialisation Produce
innovation Development (RD))
32
Open innovation
33
Open innovation
34
Open innovation
OECD Science, Technology and Industry Scoreboard,
2013
35
Open innovation
OECD Science, Technology and Industry Scoreboard,
2013
36
Open innovation
OECD Science, Technology and Industry Scoreboard,
2013
37
Relationship between research, develoment and
innovation
Overlapping area
Commercialisation
Basic research
Intellectual property rights
INNOVATION
RD
Experimental development
Technical assistance
Distribution
Applied research
Industralisation
Production
38
What is a Manual?

• Manuals are codified knowledge
• They are guidelines for the collection and
  interpretation of data and for international
  comparisons of data, statistics and indicators.
• They are supported by an international
  infrastructure (ISIC, ISCED and ISCO)
• Manuals provide a language of discourse and they
  behave like a technology (our guiding assumption)

39
The OECD Frascati Manual family

• The Measurement of Scientific and Technical
  Activities Proposed Standard Practice for
  Surveys of Research and Development. Frascati
  Manual (2002)
• Proposed Guidelines for Collecting and
  Interpreting Technological Innovation Data Oslo
  Manual (2005)
• Patent Statistics Manual (1994)
• Manual on the Measurement of Human Resources in
  Science and Technology Canberra Manual (1995)
• Proposed Standard Practice for the Collection and
  Interpretation of Data on the Technological
  Balance of Payments (1992)
• _______________________________________________
• Bibliometrics Manual

40
The OECD Frascati manual family
41
Other related manuals and classifications

• System of National Accounts SNA (CEC et al.,
  1994)
• International Standard Industrial Classification
  ISIC Rev. 3.1 (UN, 2002)
• Statistical Classification of Economic Activities
  in the European Community NACE Rev. 1.1
  series 2E.
• Nomenclature for the analysis and comparison of
  scientific programmes and budgets (NABS 2007)

42
The Frascati Manual
43
Freeman the first edition of the Frascati Manual
(1962)

• Freeman proposed standardised definitions,
  concepts, methodologies for conducting RD
  surveys and measuring inputs, namely money
  devoted to RD and ST personnel.
• Four topics
• - definition of research (basic, applied,
  development)
• - demarcation with other ST activities
  (teaching, production, etc.),
• - economic sectors (university, government,
  industry, pnp)
• - surveying methodology
• The underlying model linear

44
The RD data
People

• Money
• Expenditure (GERD)
• Budget appropriations (GBAORD)

45
Frascati Manual The definition of RD

• Research and experimental development (RD)
  comprise creative work undertaken on a systematic
  basis in order to increase the stock of
  knowledge, including knowledge of man, culture
  and society, and the use of knowledge to devise
  new applications
• Basic research
• Applied research
• Experimental development

46
Frascati Manual the definition of RD personnel

• RD personnel All persons employed directly on
  RD should be counted, as well as those providing
  direct services such as RD managers,
  administrators, and clerical staff
• Researchers
• Technicians and equivalent staff
• Other supporting staff

47
Oslo Manual

• Harmonised questionnaire

48
Types of innovation

• Product innovation (goods and services)
• Process innovation
• Marketing innovation
• Organisational innovation

49
Types of innovation

• A product innovation is the introduction of a
  good or service that is new or significantly
  improved with respect to its characteristics or
  intended uses. This includes significant
  improvements in technical specifications,
  components and materials, incorporated software,
  user friendliness or other functional
  characteristics.
• A process innovation is the implementation of a
  new or significantly improved production or
  delivery method. This includes significant
  changes in techniques, equipment and/or software
• A marketing innovation is the implementation of
  a new marketing method involving significant
  changes in product design or packaging, product
  placement, product promotion or pricing.
• An organisational innovation is the
  implementation of a new organisational method in
  the firms business practices, workplace
  organisation or external relations.

50
Types of innovation
51
Innovation the system approach

• A systems approach
• Actors Governments, education and health
  institutions, business, foreign institutions
• Activities RD, invention, innovation, diffusion
  of technologies and practices, HR development
• Linkages Contracts, collaborations,
  co-publication, grants, monitoring
• Outcomes Wealth, growth, jobs
• Impacts Wellbeing, culture change, global
  influence
• The activity of innovation is dynamic, complex,
  non-linear and global

52
Use of indicators

• Monitoring
• Benchmarking
• Foresight
• Evaluation
• Research

53
Use of indicators

• Monitoring Comparing the values of a set of
  indicators over time
• Some questions
• How much does the government spend on STI?
• Where does it spend it (geography and industry)?
• Why does it spend it (socio-economic objectives)?
• What does the government get for spending this
  money?
• Some Publications
• European Innovation Scoreboard
• OECD STI Scoreboard
• OECD Science, Technology and Industry Outlook

54
Use of indicators

• Monitoring Comparing the values of a set of
  indicators over time
• Indicators
• Innovation
• RD
• Capital investment
• Intellectual property rights
• Learning
• Education
• Design

55
Use of indicators

• Benchmarking
• 1. Decide upon a set of indicators which are
  relevant to policy objectives, and then decide
  about targets to be achieved
• 2. Agree on a set of indicators and then select
  another system which is performing better
• Examples
• Ranking of countries and institutions
• The 3 Lisbon target and ensuing policies (focus
  the public debate)

56
RD intensity in Italy the Fata Morgana of 3
3,0
1,5
1.2

• 1970

• 1990

• 1980

• 2010

• 2020

• 1960

• 2000

• 2030

• Dagli anni 60 ad oggi il rapporto RS/Pil è
  rimasto invariato, intorno all1,1. Si può
  prevedere che nei prossimi anni rimarrà agli
  stessi livelli o che possa aumentare lievemente
  (l1,5 teoricamente possibile è rimasto una
  chimera). Lobiettivo di Lisbona e Barcellona è
  stato clamorosamente mancato non soltanto in
  Italia, ma anche in Europa, anche se è stato
  riconfermato dai capi di governo per il prossimo
  futuro.

57
Use of indicators

• Foresight viewing the future involving a mix of
  quantitative and qualitative evaluation methods,
  including a set of a current set of indicators.
  Discern likely paths for emerging technologies,
  possible futures for an economic region, or
  alternative responses to a coming problem such as
  the country should function when the oil runs
  out.
• Indicators provide a background to the process
  but in general are too aggregated

58
The use of indicators

• Evaluation concerns the effective and efficient
  use allocation of resources in order to achieve a
  set of objectives
• Various methods
• Quantitative (bibliometric analysis, turnover
  from new products, audits, etc.) and qualitative
  (peer review)
• Research Assessment Exercise in the UK (now
  Research Excellence Framework VEDERE)

59
Composite indicators
60
EU Summary Innovation Index Composite indicator
(25 indicators)
3
25
61
The EU Summary Innovation Index
62
The EU innovation performance indicator
Summary Innovation Index
63
Summary Innovation Index (ctd)
64
Summary Innovation Index for EU countries
65
Profile of Italy
66
Global innovation performance
67
Innovation performance of Russia
68
Innovation performance of Russia
69
The Stiglitz report
In February 2008, the President of the French
Republic, Nicholas Sarkozy, asked Joseph Stiglitz
, Amartya Sen, and Jean Paul Fitoussi to create a
Commission, subsequently called The Commission
on the Measurement of Economic Performance and
Social Progress. The Commissions aim has been
to identify the limits of GDP as an indicator of
economic performance and social progress,
including the problems with its measurement to
consider what additional information might be
required for the production of more relevant
indicators of social progress to assess the
feasibility of alternative measurement tools, and
to discuss how to present the statistical
information in an appropriate way.
70
"Gdp and beyond" Equitable and sustainable
well-being
12 domains 1.Health 2.Education and
training 3.Work and life balance 4.Economic
well-being 5.Social relationships 6.Politics and
Institutions 7.Security 8.Subjective
well-being 9.Landscape and cultural
heritage 10.Environment 11.Research and
innovation 12.Quality of services
71
"Gdp and beyond" Equitable and sustainable
well-being

• Features of the composite indicator
• Comparisons across space
• Comparison across time
• Non-substituability on elementary indicators
• Simplicity and transparency in calculation
• Immediate use and easy interpretation of results
• Robustness
• Methodology
• Corrected MPI, Mazziotta M., Pareto A. (2013).
  A Non-compensatory Composite Index for Measuring
  Well-being over Time. Cogito. MultidisciplinaryRes
  earch Journal, vol. V, n. 4

72
Equitable and sustainable well-being
1. Intensity of research Percentage of RD
expenditure on GDP. 2. Propensity to patent
Number of patent applications filed to the
European Patent Office (EPO) per million of
inhabitants.. 3. Impact of knowledge workers on
employment Percentage of persons employed with
university education (ISCED 5-6) in
scientific-technological occupations (ISCO 2-3)
on total persons employed. 4. Innovation rate of
the productive system Percentage of firms that
have introduced technological (product or
process), organizational or marketing innovation
in a three-year period on total number of firms
with at least 10 number of persons employed.
73
Equitable and sustainable well-being (ctd)
5. Innovation rate of product/service of the
national productive system Percentage of firms
that have introduced innovations of
product/service in a three-years period on total
number of firms with at least 10 number of
persons employed . 6. Productive specialization
in knowledge-intensive sectors Percentage of
persons employed in high-tech manufacturing
sectors and those in knowledge-intensive services
on total of persons employed. 7. Intensity of
Internet use Percentage of people aged 16-74
years who have used internet at least once a week
during the 12 months before the interview on
total people aged 16-74 years.
74
Ranking of universities
Four major sources ARWU Shangai (Shangai, Jiao
Tong University) QS World University Ranking THE
University Ranking (Times Higher Education) US
News e World Reports (Best Global
Universities)
75
Academic Ranking of World Universities Shanghai
Jiao Tong University

• Starting from 2003, ARWU has been presenting the
  world Top 500 universities annually based on a
  set of objective indicators and third-party data.
  ARWU has been recognized as the precursor of
  global university rankings and the most
  trustworthy league table.
• ARWU adopts six objective indicators to rank
  world universities, including
• the number of alumni and staff winning Nobel
  Prizes and Fields Medals,
• the number of Highly Cited Researchers,
• the number of articles published in journals of
  Nature and Science,
• the number of articles indexed in Science
  Citation Index - Expanded and Social Sciences
  Citation Index, and
• per capita performance.
• More than 1200 universities are actually ranked
  by ARWU every year and the best 500 universities
  are published.

76
TopUNIVERSITIES Worldwide university rankings,
guides events
Criteria selected as the key pillars of what
makes a world class university Research Teaching
Employability Internationalisation Facilities
Social Responsibility Innovation Arts Culture
Inclusiveness Specialist Criteria
77
TopUNIVERSITIES Worldwide university rankings,
guides events
Research Indicators considered here include
assessments of research quality amongst
academics, productivity (i.e. number of papers
published), citations (i.e. how recognized and
referred to those papers are by other academics)
and awards (e.g. Nobel Prizes or Fields
Medals). Teaching A key role of a university is
the nurture of tomorrow's finest minds, inspiring
the next generation of potential research
academics. Typical indicators in teaching quality
assessments are collation of student feedback
through national student surveys, further study
rate and student faculty ratio. Employability Grad
uate employability encompasses more than academic
strength, focusing on work-readiness - the
ability to work effectively in a multi-cultural
team, to deliver presentations, to manage people
and projects. Common indicators in this area are
surveys of employers, graduate employment rates
and careers service support. Internationalisation
Here, effective indicators could be the
proportion of international students and staff,
the numbers of exchange students arriving and
departing, the number of nationalities
represented in the student body, the number and
strength of international partnerships with other
universities and the presence of religious
facilities. Facilities University infrastructure
is an indicator which enables students to know
what to expect from their university experience.
Indicators such as sporting, IT, library and
medical facilities, as well as the number of
students societies are considered within this
criterion. Online/Distance learning This category
looks at various indicators such as student
services and technology, track record, student
faculty engagement, student interaction,
commitment to online and reputation of the
university. Social Responsibility Engagement
measures how seriously a university takes its
obligations to society by investing in the local
community as well as in charity work and disaster
relief. It also analyses the regional human
capital development and environmentally
awareness. Innovation Innovation, the output of
the universities activities and findings to
economy, society and culture, has become
increasingly relevant for universities. Arts
Culture Effective indicators are the number of
concerts and exhibitions organized by the
institution, the number of credits and cultural
awards and cultural investment. Inclusiveness This
area looks at the accessibility of the
university to students, particularly at
scholarships and bursaries, disability access,
gender balance and low-income outreach. Specialist
Criteria Excellence in a narrow field is as
valid a claim to world-class status as competence
in the round. These criteria are designed to
extend credit where it's due. This category looks
at accreditations and discipline rankings.
78
Ranking of universities the case of Italy
ARWU Shangai (Shangai, Jiao Tong University) QS
World University Ranking THE University Ranking
(Times Higher Education) US News e World Reports
(Best Global Universities) ARWU Shangai
Bologna 173, Milano 186, Padova 188, Pisa 190,
Sapienza 191 QS World University Ranking
Bologna 182, Sapienza 202, Politecnico Milano
229 World University Ranking SA Sapienza 95,
Bologna 99, Pisa 184, Milano 193 US News e World
Report Sapienza 139, Bologna 146, Padova 146,
Milano 155
79
OECD Science, Technology and Industry Outlook,
2008
80
OECD Science, Technology and Industry Outlook,
2008
81
OECD Science, Technology and Industry
Scoreboard, 2013
82
OECD Science, Technology and Industry Outlook,
2008
83
OECD Science, Technology and Industry
Scoreboard, 2013
84
OECD Science, Technology and Industry Outlook,
2008
85
OECD Science, Technology and Industry
Scoreboard, 2013
86
OECD Science, Technology and Industry Outlook,
2008
87
OECD Science, Technology and Industry
Scoreboard, 2013
88
OECD Science, Technology and Industry
Scoreboard, 2013
89
OECD Science, Technology and Industry
Scoreboard, 2013
90
Clusters
91
Use of indicators (Benoit Godin)

• As OECD admitted Monitoring and benchmarking
  are not coupled with policy evaluation (). They
  are seldom used for evaluation purposes () but
  to analyse counties position vis-à-vis
  competing countries and to motivate adaptation or
  more intense policy efforts ().
• Official statistics mainly served discourse
  purposes, and in this sense the accounting
  framework and the statistics presented within it
  were influential because they fit perfectly well
  with the policy discourse on rationality,
  efficiency and accountability it aligns and
  frames the science system, by way of statistics,
  as goal-oriented and accountable. As it actually
  is, the accounting in official statistics on
  science is a metaphor, not an accounting exercise
  as such

92
A rhetoric device a plethora of figures and
graphs

• In the various studies on productivity and the
  New Economy the OECD constantly reminded the
  reader that the links between science, technology
  and productivity have not been demonstrated.
• A large series of graphs and figures could
  persuade the reader of the seriousness of the
  study. Although no statistics could be used to
  prove the emergence of the New Economy, graphs
  and figures nevertheless served the purpose of
  empiricism.
• (Godin, 2004)

93
Use of indicators (Benoit Godin)

• The rhetoric of numbers
• Policy prescriptions based on shaking statistical
  evidence (New Economy)
• The umbrella concept, slogans, buzzwords, which
  shape new ways to arrange old indicators

94
A rhetoric device a plethora of figures and
graphs

• Secure a quantitative statement of the critical
  elements in an officials problem, draw it up in
  concise form, illuminate the tables with a chart
  or two, bind the memorandum in an attractive
  cover tied with a neat bow-knot (). The data
  must be simple enough to be sent by telegraph and
  compiled overnight
• (Mitchell, 1919)

95
The mystique of ranking

• GERD is used for target setting - from
  descriptive to prescriptive
• The American GERD/GDP ratio of the early 1960s,
  that is 3, as mentioned in the first paragraphs
  of the first edition of the Frascati Manual,
  became the ideal to which member countries would
  aim, and which the OECD would implicitly promote
  
• (Godin)
• Lisbona UE 3 (2 business, 1 public sector)

96
OECD Science, Technology and Industry Outlook,
2008
97
Keith Pavitt

• One would think that the political agenda
  determines the collection and analysis of
  indicators. In reality it is the other way round
  it is the availability of indicators which steers
  the political discourse.

98
Fred Gault

• Policy analysts should be both literate and
  numerate, able to put a case using innovation
  indicators. Not only should the analysits have
  such a skill set, but they also require some
  knowledge of the subject. It is in this
  environment that monitoring, benchmarking and
  evaluation lead to policy learning and to more
  effective policies.

99
The policy advisor

• Advising has become the preserve of applied
  economists (no more physicists and sociologists)
• Advisors need to be and to appear experts who
  guarantee a competent and independent approach
• The compromise between engagement and integrity
  of analysts
• Frustration of advisors is part of the game
• The paradox too many or too little indicators?
• The pressure of vested interests on official
  statisticians

100
Innovation expenditure. The Oslo Manual
RD 38
RD 55
101
A dilemma for users

• BERD (Frascati), Annual, all NACE industries, no
  size threshold
• CIS (Oslo), selected NACE industries, 10
  employees or more, sample

102
Research Evaluation

• Evaluation of what
• research
• education
• third mission of universities and research
  agencies (consultancy, support to local
  authorities, etc.)
• Evaluation by whom
• experts, peers
• Evaluation of what
• organisations (departments, universities,
  schools)
• programmes, projects
• individuals (professors, researchers, students)
• Evaluation when
• ex-ante
• in-itinere
• ex-post

103
Research Evaluation

• Indicators used
• - bibliometrics
• - RD
• - peer review
• - students
• - graduates
• - patents
• - spin-offs
• - contracts and other funding
• - other

104
Evaluation in Italy

• New public management
• Accountability
• Value for money

105
San Francisco Declaration on Research Assessment

• The Journal Impact Factor, as calculated by
  Thomson Reuters, was originally created as a tool
  to help librarians identify journals to purchase,
  not as a measure of the scientific quality of
  research in an article.
• With that in mind, it is critical to understand
  that the Journal Impact Factor has a number of
  well-documented deficiencies as a tool for
  research assessment. These limitations include
• A) citation distributions within journals are
  highly skewed
• B) the properties of the Journal Impact Factor
  are field-specific it is a composite of
  multiple, highly diverse article types, including
  primary research papers and reviews
• C) Journal Impact Factors can be manipulated (or
  gamed) by editorial policy and
• D) data used to calculate the Journal Impact
  Factors are neither transparent nor openly
  available to the public.

106
San Francisco Declaration on Research Assessment

• General Recommendation
• Do not use journal-based metrics, such as
  Journal Impact Factors, as a surrogate measure of
  the quality of individual research articles, to
  assess an individual scientists contributions,
  or in hiring, promotion, or funding decisions.

107
Lessons from research evaluation

• Evaluation should enhance efficiency and
  effectiveness
• Evaluation is a difficult process
• Peer review vs bibliometrics
• NSE vs SSH
• Used to define financing and hiring/promoting
  staff
• League tables
• Pro-active evaluation vs punitive evaluation
• Competition vs cooperation of scientists
• Excellence
• Opportunistic behaviour and the split of the
  academic community
• Threat t othe equilibrium amongst the activities
  of the university
• Research Assessment Excercise (REF) in the UK

108
Concluding remarks

• After 50 years STI indicators are quite good
• International cooperation is key (the role of the
  OECD and EU)
• Support to non-OECD countries
• Further develop theoretical models and
  methodologies
• From macro to micro
• Keep the dialogue between actors alive
• More expertise too many economists
  (philosophers, sociologists, other social
  scientists)
• Too many indicators?
• Resource constraints
• STI indicators are a public good, and the owner
  of statistics is the State
• The role of commercial data producers (e.g.
  bibliometrics)
• What to do with the misuse of indicators?
• Producers stay creative, autonomous, and
  risk-taking

109

• Thank you for your attention
• giorgio.sirilli_at_cnr.it