Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool.

1
Bibliometric analyses of publications and grant
productivity SPIRES, a new Web-based tool.

• Todays discussion points
• Background
• - Program Analysis Branch Goals
• - Current methods for evaluating research
  including bibliometric analysis
• Introduction to SPIRES (Scientific Publications
  Information REtrieval System)
• Results of a Validation Experiment
• Future Directions/
• Live demo

2
PAB Analysis Goals

• Information transfer Communicate the
  high-quality research produced by the NIEHS
  portfolio.
• Track current DERT supported research Examine
  number of grants and annual support broken down
  into major fields of science.
• Analyze all incoming grants Develop a two-three
  sentence description of each funded grant, place
  into a web searchable data base. Track study
  sections and priority scores.
• Evaluate scientific and public health impact of
  grants. Track publications for each grant award
  and center. Every RFA and PA needs to be tracked
  for publication productivity and scientific
  impact.

3
Evaluation Tools

• peer review accepted gold standard of
  evaluation, provides a review of quality of
  research
• usually applies to specific projects, difficult
  to use for the evaluation of programs or fields,
  time consuming, expensive and labor intensive
• bibliometric analysis examines publications,
  impact factors, literature citations, and patent
  counts is based on the premise that a
  publication represents quality research as it
  passed peer review, and cited literature has
  merit
• quantitative can measure aggregate quality of
  fields or programs, but not always useful across
  fields

4
Needs for tracking publications

• To systematically track the productivity of the
  portfolio, specific tools are needed by the
  Program Analysis Branch (PAB) that are not
  currently available.
• According to NLM, NIEHS-supported grantees
  published 1637 publications in 2000.
• The idea was to link the grant database (IMPAC
  II) to the National Library of Medicine (NLM)
  database.
• Web-based routines will allow NIEHS staff to
  display information and generate multiple
  outputs.
• System should be robust so that new NLM downloads
  can be accepted and operator updates are
  feasible.

5
Development of SPIRES joining IMPAC II with NLM.

• SPIRES
  Scientific Publication
  Information REtrieval System
• NLM keeps all its data in flat files.
• Jerry Nehls downloaded ES/NIEHS data from NLM to
  a local server.
• Paul Jordan developed computer algorithms to
  search publications for Grant numbers, (ES-----)
  and developed tables in Oracle to create a
  relational database linking IMPAC II with NLM
  data.
• Bob Hoppin developed a Web-based interface.
• Paul Jordan and Ying Gao currently upgrading
  functionality and usability.

6
Status of SPIRES. Scientific Publication
Information REtrieval System

• Currently have 1995-2001 (August) data loaded
  with 9,615 publications
• SPIRES is being used by the PAB
• Finding and removing bugs
• Working on developing a better interface and
  reporting structure
• SPIRES is accessible to anyone in NIEHS
  http//carlton.niehs.nih.gov/SPIRES/PRIV/query/que
  ry_sel.cfm
• Working on the ambiguous Project code problem.
• ???ES007218 rarely is the prefix captured
  R01, T32, F32, P01
• ES-007218 hyphen present or zero
  missing?

7
Status of SPIRES (contin.). Scientific
Publication Information REtrieval System

• Implement and expand system through the SPIRES
  Working Group PAB, Computer Applications
  Specialists, DERT Staff, Sheila Newton, Larry
  Wright
• Running test reports looking at data quality
  through validation experiments
• SPIRES as a tool for analysis develop reports
  on Arsenic, Oxidative Stress, Cardiovascular
  disease
• Quarterly updates to SPIRES
• V 1.0 was beta-tested by several groups at NIH
• V 2.0 being developed and tested

http//carlton.niehs.nih.gov/SPIRES/PRIV/query/que
ry_sel.cfm V2.0 https//dobby.niehs.nih.gov/spire
s/priv/main/mainmenu.cfm
8
(No Transcript)
9
Crisp terms
10
Searched oxidative stress, and reactive oxygen
Crisp terms
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
Impact factor total citations/journal
articles/yr
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Total pubs 23, since 1995
24
(No Transcript)
25
Searched projects for oxidative stress, and
reactive oxygen
Crisp terms
26
Crisp terms
27
Projects
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
Publications from 1995- August of 2001- sorted by
publication count
39
SPIRES Validation Jerry Phelps

• Compare SPIRES data to information contained in
  the grant file (progress reports and competitive
  renewals)
• Grants Years
• Pilot studies MERIT n16 1995-2001
• R03 n 16(44) 1995-2001
• Larger study R01 n20 1995-2001
  P01 n2 1995-2001
• P30 n3 2000
• P42 n3 2000

40
SPIRES Validation
Pilot studies
Larger study
PAPERS
  
R AWARDS 475/593 P AWARDS
167/393 80 47
41
SPIRES Validation

• Based on these validation studies, Paul Jordan
  and Jerry Nehls have made improvements in the
  download process and decoding algorithms that
  will improve SPIRES ability to capture papers
  from PUBMED.
• PAB is following up with an analysis of the
  entire P30 program to give the Center Directors
  data that they will use to increase the number of
  papers that cite Center support.
• DERT and PAB will be contacting PIs who are not
  citing their grants in their publications.

42
SPIRES as an Analysis Tool
1. Comparisons of productivity for MERIT
nominees. 2. Productivity assessments for
competitive renewal applications. 3. Identify
high-impact papers from grantees who are visiting
NIEHS or being visited by NIEHS
leadership. 4. Program Evaluations Productivity
assessments for grant programs such as arsenic,
oxidative stress, P01, Children's centers, and
R03. 5. Customized reports in Excel spreadsheets
allow for easy sorting according to impact
factor, date, journal, PI, institution.
43
SPIRES Future Directions

• Develop better reporting features
• Integrate with Centers, Superfund, Contracts, and
  DIR databases
• Incorporate links to PubMed and ISI Citation
  reports
• Demo to other ICs, working towards building a
  NIH-wide system
• Consider incorporating SPIRES into the Quick View
  Reports (QVR) section of the
  Electronic Council Book
• Add additional tracking fields (through QVR)
  grant descriptors, disease, chemical, and organ
  systems
• Consider making a site available on Web for
  grantees

44
Special thanks to Paul Jordan, Ying Gao, Bob
Hoppin Jerry Nehls (CTB) Nancy Stegman
(CTB) Ben Van Houten, Jerry Phelps, Martha
Barnes, Edith Lee, Judy Hanson (PAB)
Thor Fjellstedt, Bill Suk, Anne Sassaman
(DERT) Larry Wright (DIR, Library) Sheila
Newton (OPPE)