Presentazione standard di PowerPoint - PowerPoint PPT Presentation


PPT – Presentazione standard di PowerPoint PowerPoint presentation | free to download - id: 7ba9a5-MmQ0O


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Presentazione standard di PowerPoint


Title: Presentazione standard di PowerPoint Author: Viarisio Enrico Last modified by: Giorgio Created Date: 11/20/2014 2:48:36 PM Document presentation format – PowerPoint PPT presentation

Number of Views:177
Avg rating:3.0/5.0
Slides: 90
Provided by: Viar1
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Presentazione standard di PowerPoint

Statistical indicators their relevance,
limitations and use in innovation studies and
policy making
Giorgio Sirilli Research Director
Statistical indicators
Statistic A numerical fact or datum, i.e. one
computed from a sample Statistical data Data
from a survey or administrative source used to
produce statistics Statistical indicator A
statistic, or combination of statistics,
providing information on some aspect of the state
of a system or of its change over time. (For
example, gross domestic product (GDP) provides
information on the level of value added in the
economy, and its change overtime is an indicator
of the economic state of the nation)
Theory and praxis
There is nothing more practical than a good
Albert Einstein
The value of science
William Gladstone
Michael Faraday
William Gladstone, then British Chancellor of the
Exchequer (minister of finance), asked Michael
Faraday of the practical value of
electricity. Gladstones only commentary was
but, after all, what use is it? Why, sir,
there is every probability that you will soon be
able to tax it.
Measuring in natural sciences
In search of the fundamental laws, the truth
CERN (European Organization for Nuclear Research)
Measuring in natural sciences
In search of the fundamental laws, the truth
Measuring in engineering
The highest accuracy possible
Ohm's law
Measuring in the social sciences
Statistical indicators
Statistics and chicken
LA STATISTICA Sai ched'è la statistica? È na'
cosa che serve pe fà un conto in generale de la
gente che nasce, che sta male, che more, che va
in carcere e che spósa. Ma pè me la statistica
curiosa è dove c'entra la percentuale, pè via
che, lì, la media è sempre eguale puro co' la
persona bisognosa. Me spiego da li conti che
se fanno seconno le statistiche d'adesso
risurta che te tocca un pollo all'anno e, se
nun entra nelle spese tue, t'entra ne la
statistica lo stesso perch'è c'è un antro che ne
magna due.
Trilussa (1871-1950)
Measuring in the social sciences
The assmption of coeteris paribus i. e. other
things being equal or held constant When you
measure a system, you change the system
Data sources
Statistical offices/agencies Private data
Science and technology indicators
After more that 50 years, what can be said about
Science and technology indicators
Oste, il vino è buono? Innkeeper, is the wine
Science and technology indicators
ST indicators are defined as a series of data
which measures and reflects the science and
technology endeavor of a country, demonstrates
its strengths and weaknesses and follows its
changing character notably with the aim of
providing early warning of events and trends
which might impair its capability to meet the
countrys needs. Indicators can help to shape
lines of argument and policy reasoning. They
can serve as checks, they are only part of
what is needed. (OECD, 1976)
Science and technology indicators
  • Indicators are a technology, a product, which
  • - governs behaviour
  • - is modified by users (outside of the producer
  • - develops in response to user needs
  • Data sources
  • Surveys, administrative data, private files, case
  • Data collection is informed by manuals
  • Data populate statistics which can be indicators
  • Decisions are taken on the basis of indicators

The linear model of innovation
Based on research Sequential Technocratic
Linear model
 L'intendance suivra   (Logistics,
supplies will come later)
The chain-linked model of innovation
Invention/ analytical project
  • Detailed
  • project
  • test

Re-design production
Distribution market
Potential market
Invenzione/ progettazione analitica
Progettazione dettagliata e test
Distribuzione e mercato
Mercato potenziale
Based on design Interactive Research is not a
pre-requisite for innovation
The triple helix model of innovation
The open innovation model
Measuring in the context of a model
Models or, at least, a scheme
Corrado Gini
The OECD manuals
A breaf history of STI policy
From patronage to public and private investment
Patronage from rulers Industrial
Revolution Between the First and the Second World
War rockets, nuclear energy, operations
research, DDT After the Second World War science
and technology policy from governments
A brief history of the measurement of STI
Second world war USA vs USSR The role of the
OECD Deeper and deeper
OECD publications in the area of STI
STI indicators
- RD - bibliometrics - innovation -
patents - TBP - human resources for ST -
peer review - students - graduates -
spin-offs - contracts and other funding - other
Science and technology policy
Report Science the Endless Frontier, 1945
Vannevar Bush
Science the Endless Frontier
Concerns Military security and Health Solution
Science policy The Government is particularly
fitted to perform certain functions, such as the
coordination and support of broad programs on
problems of great national importance Scientific
progress on a broad front results from the play
of free intellects, working on subjects of their
own choice, in the manner dictated by their
curiosity for exploration of the unknown. Freedom
of inquiry must be preserved under any plan for
Government support of science
The triangle in ST policy making
  • Policy makers design the future
  • Analysts interpret today
  • Data producers measure the past

Luigi Einaudi Italian President
Policy makers
Conoscere per deliberare Know first and then
Policy makers
  • Vannevar Bush
  • Report Science the Endless Frontier 1945
  • President Roosevelt

Policy makers
  • Why shoud we pay the
  • researchers if we make the
  • best shoes in the world?
  • Silvio Berlusconi

Culture does not provide food (Con la cultura
non si mangia) Giulio Tremonti
The dangerous business of statisticians
  • What matters is not how one fashions things, but
    what one does with them not the weapon, but the
    battle (). The making and the using of the tool
    are different things.
  • O. Spenger, Man and Technics A Contribution to
    a Philosophy of Life, 1932

ST statisticians arms producers?
  • I have no regret. Others are responsible for
    the bloodshed caused by the AK-47 machine gun. It
    is politicians fault non to be able to find
    appropriate solutions but rather to resort to
  • (M. Kalashnikov)

Data producers
  • Statistical agencies
  • Private sources

  • Number of publications
  • Number of citations
  • Impact factor
  • h-index

Use of publications for decision making
  • The case of China (SCI)
  • The case of Russia

The h-index (Jorge Eduardo Hirsch)
In 2005, the physicist Jorge Hirsch suggested a
new index to measure the broad impact of an
individual scientists work, the h-index . A
scientist has index h if h of his or her Np
papers have at least h citations each and the
other (Np - h) papers have h citations each. In
plain terms, a researcher has an h-index of 20 if
he or she has published 20 articles receiving at
least 20 citations each.
Impact factor (Eugene Fardfield)
The impact factor (IF) of an academic journal is
a measure reflecting the average number of
citations to recent articles published in that
journal. It is frequently used as a proxy for the
relative importance of a journal within its
field. In any given year, the impact factor of a
journal is the average number of citations
received per paper published in that journal
during the two preceding years. For example, if
a journal has an impact factor of 3 in 2008, then
its papers published in 2006 and 2007 received 3
citations each on average in 2008. ("Citable
items" for this calculation are usually articles,
reviews, proceedings, or notes not editorials or
letters to the editor).
Nobel laureates and bibliometrics (Boson in 2013)
Peter Ware Higgs 13 works, mostly in minor
journal, h-index 6 Francois Englert 89 works,
both in prestigious and minor journals, h-index
10 W. S. Boyle h-index 7 G. E. Smith h-index
5 C. K. Kao h-index 1 T. Maskawa h-index
1 Y. Namby h-index 17
How difficult is to evaluate!
Bruno Maksimovic Pontekorvo
Physics is a single discipline but unfortunately
nowadays phisicists belong to two differents
groups the theoreticians and the
experimentalists. If a thoretician does not
posses an extraordinary ability his work does not
make sense .For experimentalists also ordinary
peole can do a useful work (Enrico Fermi, 1931)
Science and ideology the impact on citations
Fall of the Berlin wall Berlin Nov. 1989
RD data from Frascati and Oslo do not match
BERD (Frascati), Annual, all NACE industries, no
threshold CIS (Oslo), Selected NACE industries,
10 employees or more, sample
The use of indicators
A rhetoric device a plethora of figures and
  • Secure a quantitative statement of the critical
    elements in an officials problem, draw it up in
    concise form, illuminate the tables with a chart
    or two, bind the memorandum in an attractive
    cover tied with a neat bow-knot (). The data
    must be simple enough to be sent by telegraph and
    compiled overnight
  • (Mitchell, 1919)

A rhetoric device a plethora of figures and
  • In the various studies on productivity and the
    New Economy the OECD constantly reminded the
    reader that the links between science, technology
    and productivity have not been demonstrated.
  • A large series of graphs and figures could
    persuade the reader of the seriousness of the
    study. Although no statistics could be used to
    prove the emergence of the New Economy, graphs
    and figures nevertheless served the purpose of
  • (Godin, 2004)

A rhetoric device a plethora of figures and
EU Innovation Index four groups
EU Regional Innovation Index heterogeneity
The mystique of ranking
  • GERD is used for target setting - from
    descriptive to prescriptive
  • The American GERD/GDP ratio of the early 1960s,
    that is 3, as mentioned in the first paragraphs
    of the first edition of the Frascati Manual,
    became the ideal to which member countries would
    aim, and which the OECD would implicitly promote
  • (Godin)

The mystique of 3
  • From descriptive to prescriptive
  • US 1 of GNP in the 1940s and 1950s
  • Africa 1 of GDP
  • Lisbona UE 3 (2 business, 1 public sector)
  • U.S. gt 3
  • Italy 1.56
  • .

Benoit Godin
  • Official statistics mainly served discourse
    purposes, and in this sense the accounting
    framework and the statistics presented within it
    were influential because they fit perfectly well
    with the policy discourse on rationality,
    efficiency and accountability. As it actually is,
    the accounting in official statistics on science
    is a metaphor, not an accounting exercise as

Keith Pavitt
  • One would think that the political agenda
    determines the collection and analysis of
    indicators. In reality it is the other way round
    it is the availability of indicators which steers
    the political discourse.

Fred Gault
  • Policy analysts should be both literate and
    numerate, able to put a case using innovation
    indicators. Not only should the analysits have
    such a skill set, but they also require some
    knowledge of the subject. It is in this
    environment that monitoring, benchmarking and
    evaluation lead to policy learning and to more
    effective policies.

Evaluation may be defined as an objective process
aimed at the critical analysis of the relevance,
efficiency, and effectiveness of policies,
programmes, projects, institutions, groups and
individual researchers in the pursuance of the
stated objectives. Evaluation consists of a set
of coordinated activities of comparative nature,
based on formalised methods and techniques
through codified procedures aimed at formulating
an assessment of intentional interventions with
reference to their implementation and to their
effectiveness. Internal/external evaluation
The first evaluation (Genesis)
The first evaluation In the beginning God created
the heaven and the earth. And God saw everything
that He had made. Behold, God said, it is very
good. And the evening and morning were the sixth
day. And on the seventh day God rested from all
His work. His Archangel came then unto Him
asking, God, how do you know that what You have
created is very good? What are Your criteria?
On what data do You base Your judgement? Arent
You a little close to the situation to make a
fair and unbiased evaluation? God thought about
these questions all that day and His rest was
greatly disturbed. On the eighth day, God said,
Lucifer, go to hell! (From Halcoms The Real
Story of Paradise Lost)
The neo-liberal wave of the 1980s
The new catchwords
New public management Value for
money Accountability Relevance Excellence
A brief history of evaluation
  • Research Assessment Exercise (RAE)
  • Research Excellence Framework (REF) (impact)
  • The REF will over time doubtless become more
    sophisticated and burdensome. In short we are
    creating a Frankenstein monster (Ben Martin)

The neo-liberal wave in Italy
Letizia Moratti Italian minister of education
and research
You first show that you use efficiently and
effectively the public money, then we will open
the strings of the purse
Never happened!
Renzi Scrapping old politicians
  • Rottamare i vecchi politici
  • Scrapping old politicians
  • Matteo Renzi

Contro lideologia della valutazione. LANVUR e
larte della rottamazione delluniversità
  • Model of firms management based on the
    principles of competitiveness and customer
    satisfaction (the market)
  • The catchwords
  • competitiveness
  • excellence
  • meritocracy
  • Evaluative state as the minimum state in
    which the government gives up the role of
    political responsibility and avoid the democratic
    debate in search of consensus, and rests on the
    automatic pilot of techno-administrative

Contro lideologia della valutazione. LANVUR e
larte della rottamazione delluniversità
ANVUR is much more than an administrative
branch. It is the outcome of a cultural and
political project aimed at reducing the range of
alternatives and hampering pluralism.
Changes in university life
  • The university has become at the mercy of
  • - increasing bibliometric measurement
  • - quality standards
  • - blind refereeing (someone sees you but you do
    not see him)
  • - bibliometric medians
  • - journal classifications (A, B, C, )
  • - opportunistic citing (fraud)
  • academic tourism
  • - administrative burden
  • - .

When you measure a system, you change the system
The epistemic consequences of bibliometrics-based
Interview of Italian researchers (40-65 years
old) Main results A drastic change of
researchers attitude due to the introduction of
bibliometrics-based evaluation The
bibliometrics-based evaluation has an extremely
strong normative function on scientific
practices, which deeply impact the epistemic
status of the disciplines
(T. Castellani, E. Pontecorvo, A. Valente,
Epistemological consequences of bibliometrics
Insights from the scientific community, Social
Epistemology Review and Reply Collective vol. 3
no. 11, 2014).
The epistemic consequences of bibliometrics-based
  • Results
  • 1. The bibliometrics-based evaluation criteria
    changed the way in which scientists choose the
    topic of their research
  • choosing a fashionable theme
  • placing the article in the tail of an important
    discovery (bandwagon effect)
  • choosing short empirical papers
  • 2. The hurry
  • 3. Interdisciplinary topics are hindered.
    Bibliometric evaluative systems encourage
    researchers not to change topic during their
  • 4. repetition of experiments is discuraged. Only
    new results are considered interesting

(T. Castellani, E. Pontecorvo, A. Valente,
Epistemological consequences of bibliometrics
Insights from the scientific community, Social
Epistemology Review and Reply Collective vol. 3
no. 11, 2014).
San Francisco Declaration on Research Assessment
  • The Journal Impact Factor, as calculated by
    Thomson Reuters, was originally created as a tool
    to help librarians identify journals to purchase,
    not as a measure of the scientific quality of
    research in an article.
  • With that in mind, it is critical to understand
    that the Journal Impact Factor has a number of
    well-documented deficiencies as a tool for
    research assessment. These limitations include
  • A) citation distributions within journals are
    highly skewed
  • B) the properties of the Journal Impact Factor
    are field-specific it is a composite of
    multiple, highly diverse article types, including
    primary research papers and reviews
  • C) Journal Impact Factors can be manipulated (or
    gamed) by editorial policy and
  • D) data used to calculate the Journal Impact
    Factors are neither transparent nor openly
    available to the public.

San Francisco Declaration on Research Assessment
San Francisco Declaration on Research Assessment
General Recommendation Do not use journal-based
metrics, such as Journal Impact Factors, as a
surrogate measure of the quality of individual
research articles, to assess an individual
scientists contributions, or in hiring,
promotion, or funding decisions.
The sorcerer's apprentice
Wolfgang Goete
Aprendiz feiticeiro An irresponsible person who
instigates a process or project which is unable
to control, risking to produce irreversible
The Leiden manifesto on bibliometrics
Diana Hicks
The Leiden Manifesto
Bibliometrics The Leiden Manifesto for research
Data are increasingly used to govern science.
Research evaluations that were once bespoke and
performed by peers are now routine and reliant on
metrics. The problem is that evaluation is now
led by the data rather than by judgement. Metrics
have proliferated usually well intentioned, not
always well informed, often ill applied. We risk
damaging the system with the very tools designed
to improve it, as evaluation is increasingly
implemented by organizations without knowledge
of, or advice on, good practice and
Ranking universities and research agencies
Ranking universities and research agencies
Evaluating, difficult and even dangerous .
Ranking of universities
Four major sources of ranking
ARWU Shangai (Shangai, Jiao Tong University) QS
World University Ranking THE University Ranking
(Times Higher Education) US News e World Reports
(Best Global Universities)
TopUNIVERSITIES Worldwide university rankings,
guides events
  • Criteria selected as the key pillars of what
    makes a world class university
  • Research
  • Teaching
  • Employability
  • Internationalisation
  • Facilities
  • Social Responsibility
  • Innovation
  • Arts Culture
  • Inclusiveness
  • Specialist Criteria

Global rankings cover less than 3-5 of the world
Ranking of universities the case of Italy
ARWU Shangai (Shangai, Jiao Tong University) QS
World University Ranking THE University Ranking
(Times Higher Education) US News e World Reports
(Best Global Universities)
ARWU Shangai Bologna 173,, Milano 186, Padova
188, Pisa 190, Sapienza 191 QS World University
Ranking Bologna 182,, Sapienza 202, Politecnico
Milano 229 World University Ranking SA Sapienza
95, Bologna 99, Pisa 184, Milano 193 US News e
World Report Sapienza 139, Bologna 146, Padova
146, Milano 155
The rank-ism (De Nicolao)
The rank-ism (De Nicolao)
The vice-rector of the univerisity of Pavia
declared that There are various rankings in the
world in each of them the University of Pavia
ranks in the firts 1. But it is not true.
According to three agencies Pavia is in the
following positions 371 QS World University
Rankings 251-275 Times Higher Education 401-500
Shanghai Ranking (ARWU)
Evaluation is an expensive exercise
Research Assessment Exercise (RAE) 540 million
Euro Research Excellence Framework (REF) 1
milllion Pounds (500 million)
Evaluation of the Quality of Research (VQR) 300
million Euro (ROARS) 182 million Euro (Geuna)
Rule of thumb less than 1 of RD budget
devoted to its evaluation
Evaluation is an expensive exercise
National Scientific Habilitation 126 million
Euro - Cost per application 2,300 euro -
Cost per job assigned 32,000 euro
The artifacts the Eiffel tower
Not only the art of the modern engineer, but
also the century of Industry and Science in which
we are living, and for which the way was prepared
by the great scientific movement of the
eighteenth century and by the Revolution of 1789,
to which this monument will be built as an
expression of France's gratitude."
A good example
Concluding remarks
  • A world of complexity
  • Pros and cons of quantifying
  • Pros and cons of story telling
  • Strike a balance between quantitative and

Concluding remarks
  • A world of complexity
  • Pros and cons of quantifying
  • Pros and cons of story telling
  • Strike a balance between quantitative and
  • A dream back to the Renaissance man

Sistine Chapel
Thank you for attention