An Experimental Comparison of Bibliometric Mapping Techniques

1
An Experimental Comparison of Bibliometric
Mapping Techniques

• Nees Jan van Eck, Ludo Waltman, Rommert Dekker
• Erasmus University Rotterdam, The Netherlands
• nvaneck,lwaltman,rdekker_at_few.eur.nl
• Jan van den Berg
• Delft University of Technology, The Netherlands
• j.vandenberg_at_tudelft.nl
• 10th International Conference on Science and
  Technology Indicators
• Vienna, September 18, 2008

1
1
1
1
2
Bibliometric mapping
Unit of analysis Unit of analysis Unit of analysis Unit of analysis Unit of analysis
Authors Journals Words/terms Web pages
Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure
Direct Direct Direct Direct Indirect Indirect Indirect
Jaccard Cosine Association strength Pearson correlation Cosine
Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique
Distance based Distance based Distance based Distance based Graph based Graph based Graph based
MDS VxOrd VOS Pajek Pathfinder networks
3
Bibliometric mapping
Unit of analysis Unit of analysis Unit of analysis Unit of analysis Unit of analysis
Authors Journals Words/terms Web pages
Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure
Direct Direct Direct Direct Indirect Indirect Indirect
Jaccard Cosine Association strength Pearson correlation Cosine
Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique
Distance based Distance based Distance based Distance based Graph based Graph based Graph based
MDS VxOrd VOS Pajek Pathfinder networks
4
Data sets
Unit of analysis Unit of analysis Unit of analysis Unit of analysis Unit of analysis
Authors Journals Words/terms Web pages

• Journals in economics, management, and operations
  research
• 2001-2005
• ISI categories business business, finance
  economics management and operations research
  management science
• 376 journals
• Terms in computational intelligence
• 2001-2005
• 337 terms

Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure
Direct Direct Direct Direct Indirect Indirect Indirect
Jaccard Cosine Association strength Pearson correlation Cosine
Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique
Distance based Distance based Distance based Distance based Graph based Graph based Graph based
MDS VxOrd VOS Pajek Pathfinder networks
5
Similarity measures
Unit of analysis Unit of analysis Unit of analysis Unit of analysis Unit of analysis
Authors Journals Words/terms Web pages
Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure
Direct Direct Direct Direct Indirect Indirect Indirect
Jaccard Cosine Association strength Pearson correlation Cosine

• Other names for the association strength
• probabilistic affinity index
• proximity index

Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique
Distance based Distance based Distance based Distance based Graph based Graph based Graph based
MDS VxOrd VOS Pajek Pathfinder networks
6
Mapping techniques
Unit of analysis Unit of analysis Unit of analysis Unit of analysis Unit of analysis
Authors Journals Words/terms Web pages

• Ordinal MDS in SPSS PROXSCAL
• VxOrd (DrL)
• VxOrd1 top 10 similarities, edge cutting
  (default settings)
• VxOrd2 all similarities, no edge cutting
• VxOrd3 top 10 similarities, no edge cutting

Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure Similarity measure
Direct Direct Direct Direct Indirect Indirect Indirect
Jaccard Cosine Association strength Pearson correlation Cosine
Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique Mapping technique
Distance based Distance based Distance based Distance based Graph based Graph based Graph based
MDS VxOrd VOS Pajek Pathfinder networks
7
Evaluation criteria
Quality of the maps
Does a map provide a satisfactory representation of the data?
Robustness
Measures the similarity between maps that have been constructed based on different samples drawn randomly from the same data set
Efficiency
Measures the similarity between a map that has been constructed based on an entire data set and maps that have been constructed based on samples drawn randomly from that data set
8
MDS Jaccard index (journals)
9
VxOrd1 cosine (journals)
10
VxOrd2 cosine (journals)
11
VxOrd3 cosine (journals)
12
VOS association strength (journals)
13
Comparison of maps (journals)

• MDS clearly performs worst, with no good
  separation between journals from different fields
• As is typical for MDS, journals are located more
  or less uniformly distributed within a circle
• VxOrd3 and VOS perform quite well, with a clear
  clustering of journals based on their field

13
14
Robustness and efficiency (journals)

• MDS and VxOrd1 have a low robustness and
  efficiency
• The other techniques do much better

14
15
MDS Jaccard index (terms)
16
VxOrd1 cosine (terms)
17
VxOrd2 cosine (terms)
18
VxOrd3 cosine (terms)
19
VOS association strength (terms)
20
Comparison of maps (terms)
20
21
Robustness and efficiency (terms)
21
22
Conclusions
MDS VxOrd VOS
Quality of the maps locates objects more or less uniformly distributed within a circle, with no clearly separated clusters produces attractive maps with a very clear clustering, but only if appropriate parameter values are chosen produces satisfactory maps without the need to choose any parameter values

• No complete specification of VxOrd is available
  in the literature

Robustness and efficiency low depends on parameter values and data set high
22