Economia%20delle%20ICT - PowerPoint PPT Presentation

About This Presentation
Title:

Economia%20delle%20ICT

Description:

Title: Economia delle ICT Author: Secondo Rolfo Last modified by: Giorgio Created Date: 1/4/1999 3:37:08 PM Document presentation format: Lucidi Company – PowerPoint PPT presentation

Number of Views:195
Avg rating:3.0/5.0
Slides: 110
Provided by: Seco99
Category:

less

Transcript and Presenter's Notes

Title: Economia%20delle%20ICT


1
Moscow January 21, 2015 An overview of STI
statistics and indicators Giorgio
Sirilli ISCrES - CNR
2
Outline of the presentation
  • STI indicators
  • STI policy
  • The actors
  • Models of innovation
  • Manuals
  • Uses and abuses of indicators
  • Concluding remarks

3
Science, technology and innovation
  • Science is the reasoned investigation or study of
    phenomena, aimed at discovering enduring
    principles among elements of the phenomenal world
    by employing formal techniques such as the
    scientific method
  • Technology is the collection of tools, including
    machinery, modifications, arrangements and
    procedures used by humans
  • Innovation is the process practical application
    of knowledge translating an idea or invention
    into a good or service that creates value or for
    which customers will pay

4
Indicators
  • Statistic A numerical fact or datum, i.e. one
    computed from a sample.
  • Statistical data Data from a survey or
    administrative source used to produce statistics.
  • Statistical indicator A statistic, or
    combinations of statistics, providing information
    on some aspect of the state of a system or of its
    change over time. (For example, gross domestic
    product (GDP) provides information on the level
    of value added in the economy, and its change
    overtime is an indicator of the economic state of
    the nation.)

5
Indicators
  • Indicators are a technology, a product, which
  • - governs behaviour
  • - is modified by users (outside of the producer
    community)
  • - develops in response to user needs
  • Data sources
  • Surveys, administrative data, private files, case
    studies
  • Data collection is informed by manuals
  • Data populate statistics which can be indicators
  • Decisions are taken on the basis of indicators

6
Science and technology indicators
  • ST indicators are defined as a series of data
    which measures and reflects the science and
    technology endeavor of a country, demonstrates
    its strengths and weaknesses and follows its
    changing character notably with the aim of
    providing early warning of events and trends
    which might impair its capability to meet the
    countrys needs.
  • Indicators can help to shape lines of argument
  • and policy reasoning. They can serve as
  • checks, they are only part of what is needed.
  • (OECD, 1976)

7
Lord Kelvin
  • If you can not measure it, you can not improve
    it.
  • Lord Kelvin

8
Hariolf Grupp
  • If you can not measure it, dont talk about it.
  • Ariolf Grupp

9
The evolution of STI indicators
  • The first attempt to measure ST in 1957
  • Frascati Manual (1963)
  • The Frascati manual family
  • A continous process of broadening and deepening
    from macro to micro, from public to private
  • The role of international organisations
  • The dialogue between producers and users

10
Science and technology policy
A history which starts after WW2
1940s
2010s
competitiveness
social needs
innovation
science
technology
jobs
11
Science and technology policy
  • Report Science the Endless Frontier 1945
    (Vannevar Bush)

12
Science the Endless Frontier
  • Concerns Military security Health
  • Solution Science policy
  • The Government is particularly fitted to perform
    certain functions, such as the coordination and
    support of broad programs on problems of great
    national importance
  • Scientific progress on a broad front results
    from the play of free intellects, working on
    subjects of their own choice, in the manner
    dictated by their curiosity for exploration of
    the unknown. Freedom of inquiry must be preserved
    under any plan for Government support of science

13
Science the Endless Frontier
  • Publicly and privately supported colleges and
    universities and the endowed research institutes
    must furnish both the new scientific knowledge
    and the trained research workers. It is chiefly
    in these institutions that scientists may work in
    an atmosphere which is relatively free from the
    adverse pressure of convention, prejudice, or
    commercial necessity. At their best they provide
    the scientific worker with a strong sense of
    solidarity and security, as well as a substantial
    degree of personal intellectual freedom.
  • Industry is generally inhibited by preconceived
    goals, by its own clearly defined standards, and
    by the constant pressure of commercial necessity.
    Satisfactory progress in basic science seldom
    occurs under conditions prevailing in the normal
    industry laboratory.

14
The triangle in STI policy making
  • Policy makers design the future
  • Analysts interpret today
  • Data producers measure the past

15
Luigi EinaudiItalian President
Policy makers
Conoscere per deliberare Know first and then
sanction
16
Policy makers
  • Vannevar Bush
  • Report Science the Endless Frontier 1945
  • President Roosevelt

17
Policy makers
  • Why shoud we pay the
  • researchers if we make the
  • best shoes in the world?
  • Silvio Berlusconi

Culture does not provide food (Con la cultura
non si mangia) Giulio Tremonti
18
The relationship between producers and users
  • After WWII the main issue in STI policy was the
    social responsibility of science. Now the
    attention is placed on the social return of
    investment on STI and, in particular, on
    innovation and social objectives
  • Weak innovation theories do not allow a
    straighforward interpretation of indicators
  • A tension the simplification of policy makers
    (e.g. the 3 RD/GDP ratio) versus deepening of
    analysts
  • The time dimension demand of indicators and
    analyses (quick and dirty) but . the
    construction of indicators is a heavy ship
    (more than 10 years to build)

19

Producers of indicators
  • OECD Group of National Experts on Science and
    Technology Indicators - NESTI

20
NESTI
  • NESTI Working Party of National Experts on
    Science and Technology Indicators (1962)
  • Mission
  • Produce methodologies, statistics and analyses
  • Co-ordinating body
  • Clearing house

21
International organisations active in STI
indicators
  • OECD and member countries
  • Eurostat and the relevant Commission DGs and
    member states of the EU
  • UNESCO
  • United Nations Institute of Statistics
  • RICYT and member countries
  • NEPAD is moving in this direction

22
The role of OECD in the development of ST
measurement in the 1960s
  • In the 1960s few countries collected data on ST
  • The OECD offered a ready-made model for those who
    had not yet developed the necessary instruments
  • Standardisation was proposed by an international
    organisation and not by a specific country
  • The Frascati Manual was introduced with a petits
    pas strategy
  • - the first edition was an internal document
    only (1962)
  • - the Manual was tested (1963-1964) in many
    countries
  • - it was revised in light of the experience
    gained from the surveys
  • The Frascati Manual has never been an imperative
    document countries are totally free to apply its
    conventions
  • No harmonised questionnaire

23
United Nations Fundamental Principles of Official
Statistics
  • Official statistics provide an indispensable
    element in the information system of a democratic
    society. They need to be compiled and made
    available on an impartial basis by official
    agencies.
  • To retain trust in official statistics, the
    statistical agencies need to decide according to
    strictly professional considerations on methods
    and procedures.
  • To facilitate a correct interpretation of the
    data, the statistical agencies are to present
    information according to scientific standards on
    the sources, methods and procedures of the
    statistics.

Source UN Statistical commission
24
United Nations Fundamental Principles of Official
Statistics
  • Data for statistical purposes may be drawn from
    all types of sources, be they statistical surveys
    or administrative records.
  • Individual data collected by statistical agencies
    are to be strictly confidential and used
    exclusively for statistical purposes.
  • The laws, regulations and measures under which
    the statistical systems operate are to be made
    public.
  • Coordination among statistical agencies is
    essential.

Source UN Statistical commission
25
The dangerous business of statisticians
  • What matters is not how one fashions things, but
    what one does with them not the weapon, but the
    battle (). The making and the using of the tool
    are different things.
  • O. Spenger, Man and Technics A Contribution to
    a Philosophy of Life, 1932

26
Indicators community
  • The members of the indicators community have to
    play the difficult game of finding a proper
    balance between data collection, analysis and
    policy making preserving their intellectual
    integrity

27
Models of innovation
  • Indicators make sense in the context of models

Linear Chain-linked Triple helix Open innovation
28
The Linear model of innovation
Research
Production
Development
Design
Engineering
  • Based on research
  • Sequential
  • Technocratic

29
The Chain-linked model of innovation
Research
  • Ricerca

Knowledge
Invention/ analytical project
Detailed project test
Re-design production
Distribution market
Potential market
  • Invenzione/
  • progettazione
  • analitica
  • Progettazione
  • dettagliata e
  • test
  • Distribuzione
  • e mercato
  • Mercato
  • potenziale

Based on design Interactive Research is not a
pre-requisite for innovation
30
The Triple-helix model of innovation
31
The Triple-helix model of innovation
Government Government-sponsored Research
Institutes Research for public purposes Applied
research Universities Supply scientists
engineers Curiosity-driven Basic
research Firms Commercialisation Produce
innovation Development (RD))
32
Open innovation
33
Open innovation
34
Open innovation
OECD Science, Technology and Industry Scoreboard,
2013
35
Open innovation
OECD Science, Technology and Industry Scoreboard,
2013
36
Open innovation
OECD Science, Technology and Industry Scoreboard,
2013
37
Relationship between research, develoment and
innovation
Overlapping area
Commercialisation
Basic research
Intellectual property rights
INNOVATION
RD
Experimental development
Technical assistance
Distribution
Applied research
Industralisation
Production
38
What is a Manual?
  • Manuals are codified knowledge
  • They are guidelines for the collection and
    interpretation of data and for international
    comparisons of data, statistics and indicators.
  • They are supported by an international
    infrastructure (ISIC, ISCED and ISCO)
  • Manuals provide a language of discourse and they
    behave like a technology (our guiding assumption)

39
The OECD Frascati Manual family
  • The Measurement of Scientific and Technical
    Activities Proposed Standard Practice for
    Surveys of Research and Development. Frascati
    Manual (2002)
  • Proposed Guidelines for Collecting and
    Interpreting Technological Innovation Data Oslo
    Manual (2005)
  • Patent Statistics Manual (1994)
  • Manual on the Measurement of Human Resources in
    Science and Technology Canberra Manual (1995)
  • Proposed Standard Practice for the Collection and
    Interpretation of Data on the Technological
    Balance of Payments (1992)
  • _______________________________________________
  • Bibliometrics Manual

40
The OECD Frascati manual family
41
Other related manuals and classifications
  • System of National Accounts SNA (CEC et al.,
    1994)
  • International Standard Industrial Classification
    ISIC Rev. 3.1 (UN, 2002)
  • Statistical Classification of Economic Activities
    in the European Community NACE Rev. 1.1
    series 2E.
  • Nomenclature for the analysis and comparison of
    scientific programmes and budgets (NABS 2007)

42
The Frascati Manual
43
Freeman the first edition of the Frascati Manual
(1962)
  • Freeman proposed standardised definitions,
    concepts, methodologies for conducting RD
    surveys and measuring inputs, namely money
    devoted to RD and ST personnel.
  • Four topics
  • - definition of research (basic, applied,
    development)
  • - demarcation with other ST activities
    (teaching, production, etc.),
  • - economic sectors (university, government,
    industry, pnp)
  • - surveying methodology
  • The underlying model linear

44
The RD data
People
  • Money
  • Expenditure (GERD)
  • Budget appropriations (GBAORD)

45
Frascati Manual The definition of RD
  • Research and experimental development (RD)
    comprise creative work undertaken on a systematic
    basis in order to increase the stock of
    knowledge, including knowledge of man, culture
    and society, and the use of knowledge to devise
    new applications
  • Basic research
  • Applied research
  • Experimental development

46
Frascati Manual the definition of RD personnel
  • RD personnel All persons employed directly on
    RD should be counted, as well as those providing
    direct services such as RD managers,
    administrators, and clerical staff
  • Researchers
  • Technicians and equivalent staff
  • Other supporting staff

47
Oslo Manual
  • Harmonised questionnaire

48
Types of innovation
  • Product innovation (goods and services)
  • Process innovation
  • Marketing innovation
  • Organisational innovation

49
Types of innovation
  • A product innovation is the introduction of a
    good or service that is new or significantly
    improved with respect to its characteristics or
    intended uses. This includes significant
    improvements in technical specifications,
    components and materials, incorporated software,
    user friendliness or other functional
    characteristics.
  • A process innovation is the implementation of a
    new or significantly improved production or
    delivery method. This includes significant
    changes in techniques, equipment and/or software
  • A marketing innovation is the implementation of
    a new marketing method involving significant
    changes in product design or packaging, product
    placement, product promotion or pricing.
  • An organisational innovation is the
    implementation of a new organisational method in
    the firms business practices, workplace
    organisation or external relations.

50
Types of innovation
51
Innovation the system approach
  • A systems approach
  • Actors Governments, education and health
    institutions, business, foreign institutions
  • Activities RD, invention, innovation, diffusion
    of technologies and practices, HR development
  • Linkages Contracts, collaborations,
    co-publication, grants, monitoring
  • Outcomes Wealth, growth, jobs
  • Impacts Wellbeing, culture change, global
    influence
  • The activity of innovation is dynamic, complex,
    non-linear and global

52
Use of indicators
  • Monitoring
  • Benchmarking
  • Foresight
  • Evaluation
  • Research

53
Use of indicators
  • Monitoring Comparing the values of a set of
    indicators over time
  • Some questions
  • How much does the government spend on STI?
  • Where does it spend it (geography and industry)?
  • Why does it spend it (socio-economic objectives)?
  • What does the government get for spending this
    money?
  • Some Publications
  • European Innovation Scoreboard
  • OECD STI Scoreboard
  • OECD Science, Technology and Industry Outlook

54
Use of indicators
  • Monitoring Comparing the values of a set of
    indicators over time
  • Indicators
  • Innovation
  • RD
  • Capital investment
  • Intellectual property rights
  • Learning
  • Education
  • Design

55
Use of indicators
  • Benchmarking
  • 1. Decide upon a set of indicators which are
    relevant to policy objectives, and then decide
    about targets to be achieved
  • 2. Agree on a set of indicators and then select
    another system which is performing better
  • Examples
  • Ranking of countries and institutions
  • The 3 Lisbon target and ensuing policies (focus
    the public debate)

56
RD intensity in Italy the Fata Morgana of 3
3,0
1,5
1.2
  • 1970
  • 1990
  • 1980
  • 2010
  • 2020
  • 1960
  • 2000
  • 2030
  • Dagli anni 60 ad oggi il rapporto RS/Pil è
    rimasto invariato, intorno all1,1. Si può
    prevedere che nei prossimi anni rimarrà agli
    stessi livelli o che possa aumentare lievemente
    (l1,5 teoricamente possibile è rimasto una
    chimera). Lobiettivo di Lisbona e Barcellona è
    stato clamorosamente mancato non soltanto in
    Italia, ma anche in Europa, anche se è stato
    riconfermato dai capi di governo per il prossimo
    futuro.

57
Use of indicators
  • Foresight viewing the future involving a mix of
    quantitative and qualitative evaluation methods,
    including a set of a current set of indicators.
    Discern likely paths for emerging technologies,
    possible futures for an economic region, or
    alternative responses to a coming problem such as
    the country should function when the oil runs
    out.
  • Indicators provide a background to the process
    but in general are too aggregated

58
The use of indicators
  • Evaluation concerns the effective and efficient
    use allocation of resources in order to achieve a
    set of objectives
  • Various methods
  • Quantitative (bibliometric analysis, turnover
    from new products, audits, etc.) and qualitative
    (peer review)
  • Research Assessment Exercise in the UK (now
    Research Excellence Framework VEDERE)

59
Composite indicators
60
EU Summary Innovation Index Composite indicator
(25 indicators)
3
25
61
The EU Summary Innovation Index
62
The EU innovation performance indicator
Summary Innovation Index
63
Summary Innovation Index (ctd)
64
Summary Innovation Index for EU countries
65
Profile of Italy
66
Global innovation performance
67
Innovation performance of Russia
68
Innovation performance of Russia
69
The Stiglitz report
In February 2008, the President of the French
Republic, Nicholas Sarkozy, asked Joseph Stiglitz
, Amartya Sen, and Jean Paul Fitoussi to create a
Commission, subsequently called The Commission
on the Measurement of Economic Performance and
Social Progress. The Commissions aim has been
to identify the limits of GDP as an indicator of
economic performance and social progress,
including the problems with its measurement to
consider what additional information might be
required for the production of more relevant
indicators of social progress to assess the
feasibility of alternative measurement tools, and
to discuss how to present the statistical
information in an appropriate way.
70
"Gdp and beyond" Equitable and sustainable
well-being
12 domains 1.Health 2.Education and
training 3.Work and life balance 4.Economic
well-being 5.Social relationships 6.Politics and
Institutions 7.Security 8.Subjective
well-being 9.Landscape and cultural
heritage 10.Environment 11.Research and
innovation 12.Quality of services
71
"Gdp and beyond" Equitable and sustainable
well-being
  • Features of the composite indicator
  • Comparisons across space
  • Comparison across time
  • Non-substituability on elementary indicators
  • Simplicity and transparency in calculation
  • Immediate use and easy interpretation of results
  • Robustness
  • Methodology
  • Corrected MPI, Mazziotta M., Pareto A. (2013).
    A Non-compensatory Composite Index for Measuring
    Well-being over Time. Cogito. MultidisciplinaryRes
    earch Journal, vol. V, n. 4

72
Equitable and sustainable well-being
1. Intensity of research Percentage of RD
expenditure on GDP. 2. Propensity to patent
Number of patent applications filed to the
European Patent Office (EPO) per million of
inhabitants.. 3. Impact of knowledge workers on
employment Percentage of persons employed with
university education (ISCED 5-6) in
scientific-technological occupations (ISCO 2-3)
on total persons employed. 4. Innovation rate of
the productive system Percentage of firms that
have introduced technological (product or
process), organizational or marketing innovation
in a three-year period on total number of firms
with at least 10 number of persons employed.
73
Equitable and sustainable well-being (ctd)
5. Innovation rate of product/service of the
national productive system Percentage of firms
that have introduced innovations of
product/service in a three-years period on total
number of firms with at least 10 number of
persons employed . 6. Productive specialization
in knowledge-intensive sectors Percentage of
persons employed in high-tech manufacturing
sectors and those in knowledge-intensive services
on total of persons employed. 7. Intensity of
Internet use Percentage of people aged 16-74
years who have used internet at least once a week
during the 12 months before the interview on
total people aged 16-74 years.
74
Ranking of universities
Four major sources ARWU Shangai (Shangai, Jiao
Tong University) QS World University Ranking THE
University Ranking (Times Higher Education) US
News e World Reports (Best Global
Universities)
75
Academic Ranking of World Universities Shanghai
Jiao Tong University
  • Starting from 2003, ARWU has been presenting the
    world Top 500 universities annually based on a
    set of objective indicators and third-party data.
    ARWU has been recognized as the precursor of
    global university rankings and the most
    trustworthy league table.
  • ARWU adopts six objective indicators to rank
    world universities, including
  • the number of alumni and staff winning Nobel
    Prizes and Fields Medals,
  • the number of Highly Cited Researchers,
  • the number of articles published in journals of
    Nature and Science,
  • the number of articles indexed in Science
    Citation Index - Expanded and Social Sciences
    Citation Index, and
  • per capita performance.
  • More than 1200 universities are actually ranked
    by ARWU every year and the best 500 universities
    are published.

76
TopUNIVERSITIES Worldwide university rankings,
guides events
Criteria selected as the key pillars of what
makes a world class university Research Teaching
Employability Internationalisation Facilities
Social Responsibility Innovation Arts Culture
Inclusiveness Specialist Criteria
77
TopUNIVERSITIES Worldwide university rankings,
guides events
Research Indicators considered here include
assessments of research quality amongst
academics, productivity (i.e. number of papers
published), citations (i.e. how recognized and
referred to those papers are by other academics)
and awards (e.g. Nobel Prizes or Fields
Medals). Teaching A key role of a university is
the nurture of tomorrow's finest minds, inspiring
the next generation of potential research
academics. Typical indicators in teaching quality
assessments are collation of student feedback
through national student surveys, further study
rate and student faculty ratio. Employability Grad
uate employability encompasses more than academic
strength, focusing on work-readiness - the
ability to work effectively in a multi-cultural
team, to deliver presentations, to manage people
and projects. Common indicators in this area are
surveys of employers, graduate employment rates
and careers service support. Internationalisation
Here, effective indicators could be the
proportion of international students and staff,
the numbers of exchange students arriving and
departing, the number of nationalities
represented in the student body, the number and
strength of international partnerships with other
universities and the presence of religious
facilities. Facilities University infrastructure
is an indicator which enables students to know
what to expect from their university experience.
Indicators such as sporting, IT, library and
medical facilities, as well as the number of
students societies are considered within this
criterion. Online/Distance learning This category
looks at various indicators such as student
services and technology, track record, student
faculty engagement, student interaction,
commitment to online and reputation of the
university. Social Responsibility Engagement
measures how seriously a university takes its
obligations to society by investing in the local
community as well as in charity work and disaster
relief. It also analyses the regional human
capital development and environmentally
awareness. Innovation Innovation, the output of
the universities activities and findings to
economy, society and culture, has become
increasingly relevant for universities. Arts
Culture Effective indicators are the number of
concerts and exhibitions organized by the
institution, the number of credits and cultural
awards and cultural investment. Inclusiveness This
area looks at the accessibility of the
university to students, particularly at
scholarships and bursaries, disability access,
gender balance and low-income outreach. Specialist
Criteria Excellence in a narrow field is as
valid a claim to world-class status as competence
in the round. These criteria are designed to
extend credit where it's due. This category looks
at accreditations and discipline rankings.
78
Ranking of universities the case of Italy
ARWU Shangai (Shangai, Jiao Tong University) QS
World University Ranking THE University Ranking
(Times Higher Education) US News e World Reports
(Best Global Universities) ARWU Shangai
Bologna 173, Milano 186, Padova 188, Pisa 190,
Sapienza 191 QS World University Ranking
Bologna 182, Sapienza 202, Politecnico Milano
229 World University Ranking SA Sapienza 95,
Bologna 99, Pisa 184, Milano 193 US News e World
Report Sapienza 139, Bologna 146, Padova 146,
Milano 155
79
OECD Science, Technology and Industry Outlook,
2008
80
OECD Science, Technology and Industry Outlook,
2008
81
OECD Science, Technology and Industry
Scoreboard, 2013
82
OECD Science, Technology and Industry Outlook,
2008
83
OECD Science, Technology and Industry
Scoreboard, 2013
84
OECD Science, Technology and Industry Outlook,
2008
85
OECD Science, Technology and Industry
Scoreboard, 2013
86
OECD Science, Technology and Industry Outlook,
2008
87
OECD Science, Technology and Industry
Scoreboard, 2013
88
OECD Science, Technology and Industry
Scoreboard, 2013
89
OECD Science, Technology and Industry
Scoreboard, 2013
90
Clusters
91
Use of indicators (Benoit Godin)
  • As OECD admitted Monitoring and benchmarking
    are not coupled with policy evaluation (). They
    are seldom used for evaluation purposes () but
    to analyse counties position vis-à-vis
    competing countries and to motivate adaptation or
    more intense policy efforts ().
  • Official statistics mainly served discourse
    purposes, and in this sense the accounting
    framework and the statistics presented within it
    were influential because they fit perfectly well
    with the policy discourse on rationality,
    efficiency and accountability it aligns and
    frames the science system, by way of statistics,
    as goal-oriented and accountable. As it actually
    is, the accounting in official statistics on
    science is a metaphor, not an accounting exercise
    as such

92
A rhetoric device a plethora of figures and
graphs
  • In the various studies on productivity and the
    New Economy the OECD constantly reminded the
    reader that the links between science, technology
    and productivity have not been demonstrated.
  • A large series of graphs and figures could
    persuade the reader of the seriousness of the
    study. Although no statistics could be used to
    prove the emergence of the New Economy, graphs
    and figures nevertheless served the purpose of
    empiricism.
  • (Godin, 2004)

93
Use of indicators (Benoit Godin)
  • The rhetoric of numbers
  • Policy prescriptions based on shaking statistical
    evidence (New Economy)
  • The umbrella concept, slogans, buzzwords, which
    shape new ways to arrange old indicators

94
A rhetoric device a plethora of figures and
graphs
  • Secure a quantitative statement of the critical
    elements in an officials problem, draw it up in
    concise form, illuminate the tables with a chart
    or two, bind the memorandum in an attractive
    cover tied with a neat bow-knot (). The data
    must be simple enough to be sent by telegraph and
    compiled overnight
  • (Mitchell, 1919)

95
The mystique of ranking
  • GERD is used for target setting - from
    descriptive to prescriptive
  • The American GERD/GDP ratio of the early 1960s,
    that is 3, as mentioned in the first paragraphs
    of the first edition of the Frascati Manual,
    became the ideal to which member countries would
    aim, and which the OECD would implicitly promote
  • (Godin)
  • Lisbona UE 3 (2 business, 1 public sector)

96
OECD Science, Technology and Industry Outlook,
2008
97
Keith Pavitt
  • One would think that the political agenda
    determines the collection and analysis of
    indicators. In reality it is the other way round
    it is the availability of indicators which steers
    the political discourse.

98
Fred Gault
  • Policy analysts should be both literate and
    numerate, able to put a case using innovation
    indicators. Not only should the analysits have
    such a skill set, but they also require some
    knowledge of the subject. It is in this
    environment that monitoring, benchmarking and
    evaluation lead to policy learning and to more
    effective policies.

99
The policy advisor
  • Advising has become the preserve of applied
    economists (no more physicists and sociologists)
  • Advisors need to be and to appear experts who
    guarantee a competent and independent approach
  • The compromise between engagement and integrity
    of analysts
  • Frustration of advisors is part of the game
  • The paradox too many or too little indicators?
  • The pressure of vested interests on official
    statisticians

100
Innovation expenditure. The Oslo Manual
RD 38
RD 55
101
A dilemma for users
  • BERD (Frascati), Annual, all NACE industries, no
    size threshold
  • CIS (Oslo), selected NACE industries, 10
    employees or more, sample

102
Research Evaluation
  • Evaluation of what
  • research
  • education
  • third mission of universities and research
    agencies (consultancy, support to local
    authorities, etc.)
  • Evaluation by whom
  • experts, peers
  • Evaluation of what
  • organisations (departments, universities,
    schools)
  • programmes, projects
  • individuals (professors, researchers, students)
  • Evaluation when
  • ex-ante
  • in-itinere
  • ex-post

103
Research Evaluation
  • Indicators used
  • - bibliometrics
  • - RD
  • - peer review
  • - students
  • - graduates
  • - patents
  • - spin-offs
  • - contracts and other funding
  • - other

104
Evaluation in Italy
  • New public management
  • Accountability
  • Value for money

105
San Francisco Declaration on Research Assessment
  • The Journal Impact Factor, as calculated by
    Thomson Reuters, was originally created as a tool
    to help librarians identify journals to purchase,
    not as a measure of the scientific quality of
    research in an article.
  • With that in mind, it is critical to understand
    that the Journal Impact Factor has a number of
    well-documented deficiencies as a tool for
    research assessment. These limitations include
  • A) citation distributions within journals are
    highly skewed
  • B) the properties of the Journal Impact Factor
    are field-specific it is a composite of
    multiple, highly diverse article types, including
    primary research papers and reviews
  • C) Journal Impact Factors can be manipulated (or
    gamed) by editorial policy and
  • D) data used to calculate the Journal Impact
    Factors are neither transparent nor openly
    available to the public.

106
San Francisco Declaration on Research Assessment
  • General Recommendation
  • Do not use journal-based metrics, such as
    Journal Impact Factors, as a surrogate measure of
    the quality of individual research articles, to
    assess an individual scientists contributions,
    or in hiring, promotion, or funding decisions.

107
Lessons from research evaluation
  • Evaluation should enhance efficiency and
    effectiveness
  • Evaluation is a difficult process
  • Peer review vs bibliometrics
  • NSE vs SSH
  • Used to define financing and hiring/promoting
    staff
  • League tables
  • Pro-active evaluation vs punitive evaluation
  • Competition vs cooperation of scientists
  • Excellence
  • Opportunistic behaviour and the split of the
    academic community
  • Threat t othe equilibrium amongst the activities
    of the university
  • Research Assessment Excercise (REF) in the UK

108
Concluding remarks
  • After 50 years STI indicators are quite good
  • International cooperation is key (the role of the
    OECD and EU)
  • Support to non-OECD countries
  • Further develop theoretical models and
    methodologies
  • From macro to micro
  • Keep the dialogue between actors alive
  • More expertise too many economists
    (philosophers, sociologists, other social
    scientists)
  • Too many indicators?
  • Resource constraints
  • STI indicators are a public good, and the owner
    of statistics is the State
  • The role of commercial data producers (e.g.
    bibliometrics)
  • What to do with the misuse of indicators?
  • Producers stay creative, autonomous, and
    risk-taking

109
  • Thank you for your attention
  • giorgio.sirilli_at_cnr.it
Write a Comment
User Comments (0)
About PowerShow.com