Bibliometric assessment of research performance in social sciences and humanities

1
Bibliometric assessment of research performance
in social sciences and humanities

• Henk F. Moed
• Centre for Science and Technology Studies (CWTS),
  Leiden University, the Netherlands

2
Assessment of research performance Basic
assumptions - 1

• The concept of research quality does have a
  meaning in all fields of science and scholarship
• An assessment methodology should take into
  account the nature of the field

3
Assessment of research performance Basic
assumptions - 2

• Important contributions to scholarly progress are
  sooner or later communicated in scholarly
  publications

4
Research quality Dimensions

• Contribution to scholarly progress
• Enlightenment of the general public

5
Science versus humanities (Price) - 1
Science Humanities
Orientation Quantitative Qualitative
Status of knowledge Short term permanence Permanent significance
Organisation Groups interacting at the research front Individuals
6
Science versus humanities (Price) - 2

• Different substantive contents ask for different
  types of social organisation and information
  exchange
• Science Citation Index
• Humanities Normal archival library

7
Social sciences

• Social sciences constitute a heterogeneous
  domain, with both science-like and
  humanities-like orientations
• Even sub-disciplines may be heterogeneous (e.g.,
  sociology)

8
Overall ISI coverage by main field
EXCELLENT (gt 80) VERY GOOD (60-80) GOOD(40-60)
Biochem Mol Biol Appl Phys Chem Mathematics
Biol Sci Humans Biol Sci Anim Plants Economics
Chemistry Psychol Psychiat Engineering
Clin Medicine Geosciences MODERATE (lt40 )
Physics Astron Soc Sci Medicine health Other Soc Sci
Humanities Arts
9
Sub-disciplines (non-exhaustive list)
Social sciences related to medicine and health Other social sciences Humanities
Public environm occupat health Nursing Sport sciences Substance abuse Sociology Anthropology Educational sciences Political science Law Literature Linguistics Historical sc Philosophy
10
Large differences in ISI coverage among
(publishing) countries in humanities and other
social sciences
11
Other social sciences and humanities

• National publication model is dominant
• Books play an important role
• Basic assumptions of a journal citation index are
  less valid
• ISI Indexes have inadequate coverage
• They also contain national journals

12
Indicator of a journals national orientation
(INO)

• The share of the papers from the country most
  frequently publishing in a journal
• A purely national journal would have an INO value
  of 100 per cent

13
National orientation of journals (INO) for 4
fields
Discipline No. Journals Median INO Journals with INOgt90
Physics astronomy 260 37 10
Mol Biol biochem 530 41 4
Humanities arts 1,110 71 24
Other social sci 879 72 22
14
Policy assumption for social sciences and
humanities

• The extent to which research findings
• reach beyond a purely national or local viewpoint
• and are exposed to criticisms from a wide
  international scholarly audience
• is a relevant criterion of research quality

15
The problem ISI ? International

• Works exposed to an international audience are
  not necessarily included in the ISI Indexes
• Works included in the ISI Indexes are not
  necessarily exposed to an international audience

16
This book argues

• It cannot be taken for granted that the ISI
  Citation Indexes provide such indicators in all
  subfields of these domains of scholarship
• A challenge would be to systematically explore
  alternative data sources and methodologies

17
4 Types of bibliometric studies
Cited/Target Citing/Source ISI coverage Field/Study
1 ISI ISI Excellent Very Good Astronomy
2 ISInon ISI ISI Very Good Good Mathematics
3 ISInon ISI ISInon ISI Good Moderate Computer Science
4 No citation analysis at all No citation analysis at all Moderate NL Law
18
Alternative approaches

1. Expand the WoS with additional sources
2. Classification of publications and sources based
   on scholars quality perceptions
3. Library collection analysis

19
1. Expand the WoS with additional sources

• University of Granada (Spain)
• Creation of a citation index with 200 Spanish
  social science source journals not covered by the
  WoS
• 50,000 uses per year

20
2. Classifications based on scholars quality
perceptions

• Case study on Flemish Law
• Questionnaires No citation analysis
• Publications in Dutch 81
• Publications in journals 59

21
Classification of (national) journals based on a
questionnaire into

• Scholarly vs. non-scholarly
• Outstanding (A), good (B) and less good (C)

22
The best indicator of scholarly research
performance (in Flemish Law)

• Count the number of
• Single and multi-authored books (first editions
  only)
• PhD theses
• Publications with a length gt 5 pages

23
3. Library collection analysis

• Focuses on books
• Determine the number of academic library copies
  per book title
• Example Use Worldcat (Linmans, CWTS, 2007)

24
END