Bibliometric indicators, research evaluation and funding parameters HEFCE Consultation Events Januar

1
Bibliometric indicators, research evaluation and
funding parametersHEFCE Consultation Events
January 2008

• Henk F. Moed
• Centre for Science and Technology Studies (CWTS)
• Leiden University, the Netherlands

2
Contents

• Short introduction to citation analysis
• Earlier studies and their outcomes
• Effects of performance assessments upon
  performers behavior
• Metrics, research evaluation and funding
  parameters

3
Citation Analysis in Research Evaluation
Springer, 2005, 350 pp.

• Henk F. Moed
• CWTS, Leiden University, the Netherlands

4
A concise introduction to citation analysis
5
Topics

• Differences in citation practices among fields
• Skewness of citation distributions
• Field-normalised citation impact indicators
• Coverage of the Web of Science
• Types of bibliometric studies
• What do citations measure

6
Scope Science vs. Technology

7
Major differences in citation levels and
half-lives (CHL) among research fields
CHL5.4 0.1
CHL8.8 0.3
CHL12.1 0.4
CHL5.0 0.1
8
(No Transcript)
9
Normalised citation impact (1.0 at world
average)

• The average citation rate of a units papers
• world citation average in the subfields in which
  the unit is active

Corrects for differences in citation practices
among fields, publication years and type of
article
10
Normalized journal impact factors
11
Profile of a group in Medical Pharmacology
Normalised citation impact
Impact BlueHigh OrangeAverage WhiteLow
Articles
12
159 NL Academic Chemistry Departments
World Average
13
Skewed citation distribution of 2 journals
N1,932 Mean4.5 Uncited12
N1,466 Mean1.9 Uncited28
14
A skewed citation distribution of a research
groupss papers is a normal phenomenon
N174 Mean5.5 Skewn4.6
N145 Mean4.8 Skewn1.6
N150 Mean38.5 Skewn2.6
72
280
15
Skewed citation distributions
Oeuvre-hypothesis

• Articles are elements from publication oeuvres of
  groups carrying out a research programme
• Authors citing an oeuvre/programme tend to cite
  key or flag papers from that oeuvre
• Key papers may snatch away citations from the
  oeuvres other articles

16
Measurement of internal WoS Coverage
Citing/Source
Non-WoS
WoS
Non-Wos Journals Books Conference
proceedings Reports Etc.
?
?
Non-WoS
Cited/Target
WoS
17
Not in WoS
WoS Coverage 5/7 71
18
Overall WoS coverage by main field
19
Three types of citation analysis
Citing/Source
Non-WoS
WoS
3. Source Expanded
2. Target Expanded
1. Pure WoS
Non-WoS
Cited/Target
WoS
20
4 Types of bibliometric studies
21
What do citations measure?

• Many studies showed positive correlations between
  citations and qualitative judgments
• In principle it is valid to interpret citations
  in terms of intellectual influence
• But the concepts of citation impact and
  intellectual influence do not coincide

22
2. Some earlier studies and their outcomes
23
16 broad disciplines
24
General European Univ
Among top 25 in publication output and
citation impact
TOP 25
Impact
BOTTOM 25
Publications
TOP 25
BOTTOM 25
25
Top US/UK research university
University has a top position in each discipline
26
(No Transcript)
27
Normalised citation impact
Impact BlackHigh DashedAverage BlankLow
Articles
28
2.5
Biol Sci Humans
29
3. Effects of performance assessments upon
performers behavior
30
Topics

• The UK Research Assessment Exercises (RAE)

31
Timing effects and shifts in criteria in UK
Research Assessment Exercises (RAE)
Total Publication Counts
Shift from Quantity to Quality
Research Active Staff
UK Articles
UK authors
1995/6
1992
2000
32
Can one increase actual citation impact by..

• Increasing author self citation?
• Publishing in high impact journals?
• Collaborate more intensively?
• Publishing with US authors because they overcite
  their own papers?
• Publishing less, only the very best papers?
• Making citation arrangements?

33
At the level of research groups, actual citation
impact and journal prestige tend to show only
weak correlationsSet of 2,150 UK authors with
gt 10 articles per year
34
More collaboration ? higher impact?

• Some studies report positive correlations between
  a papers number of authors and its citation
  impact
• They ignore differences among fields
• It all depends upon who collaborates with whom
• Causality issue Good research may lead to
  collaboration

35
Do US scientists overcite papers from their own
country?

• The crucial issue at stake is the adequacy of the
  norm against which referencing practices of US
  scientists is evaluated
• A first study found no conclusive evidence that
  US scientists in science fields excessively cite
  papers originating from their own country (Moed,
  2005)

36
Publishing less ? Higher impact?

• One would expect a higher citation impact per
  paper (crown indicator)
• But what are the longer tem effects?
• PhD students need papers in their CVs
• Relationship between a research groups bricks
  and flag papers is complex

37
Mutual citation arrangements?

• A high impact group receives its citations from
  dozens if not hundreds of institutions
• The distribution of citations amongst citing
  institutions is very skewed
• The contribution of the tail is very large
• Making arrangements with a few institutions will
  not help much

38
Why use sophisticated citation analysis in RAE?

• Shift in focus from quantity to quality
• Reduction unintended effects of using less
  sophisticated indicators
• Use of absolute rather than relative standards
• Formal rather than informal use (transparency)

39
4. Metrics, research evaluation and funding
parameters
40
Metrics and funding parameters

• Policy level Central vs. institutional (e.g.,
  national vs. university)
• Elaboration of indicator data Statistical vs.
  evaluative

41
Metrics and funding parameters
Data
Elaboration
Funding parameters
Aggregate per institution
Across institutions
Central
Indicators of groups / individuals
Within an institution
Combine with peer review
Institution
42
Use of metrics in allocation of research funds

• At a central level To distribute funds across
  institutions based on aggregate statistics
• At an institutional level Combined with peer
  review to evaluate groups and individuals
  outcomes are used to distribute funds within an
  institution

43
Aggregate statistics

• Random errors to some extent cancel out
• Identify and neutralize systematic errors

44
END