Author Classification of ARVO Conference Abstracts as Controlled Clinical Trials

1
Author Classification of ARVO Conference
Abstracts as Controlled Clinical Trials
Roberta W. Scherer1, Pamela C. Sieving2, and Kay
Dickersin1, Johns Hopkins Bloomberg School of
Public Health1, NIH Library2
RESULTS
OBJECTIVE
Author Identification of Clinical Trials by
Design Sub-classification

• 295 of 5,610 abstracts from the 2004 ARVO
  meeting, and 329 of 5,732 abstracts from the 2005
  ARVO meeting comprised the gold standard of
  abstracts describing CTs.
• Abstract authors checked the box indicating that
  they considered the abstract to describe a human
  CT in 634 abstracts in 2004, and 485 in 2005.

To test whether abstract authors could contribute
to efficient identification of controlled
clinical trial (CT) reports published in ARVO
proceedings.
BACKGROUND
2004
2005
Only about 60 of controlled clinical trial
(CTs) presented at scientific meetings and
published as abstracts in conference proceedings
are published in full (Scherer, 2005).
Accordingly, it is important to handsearch
conference proceedings to ensure comprehensive
trial identification for systematic
reviews. The annual meeting of the Association
for Research in Vision and Ophthalmology (ARVO)
is an important source of reports of clinical
trials in the field of vision yet only a small
proportion of abstracts describe CTs (e.g.,
214/5,061 (4) 2001 ARVO abstracts were
identified as CTs by a handsearcher.)
Handsearching ARVO abstracts is thus not an
efficient method of identifying CTs.
Classification as Clinical Trial by Author
Most of the differences in classification between
authors and handsearchers are among abstracts
categorized as controlled clinical trials (CCTs).
Authors identified 57.5 abstracts categorized
as CCTs in 2004 while the handsearcher identified
86.8 (92/108). In 2005, authors identified
46.2 (58/119), while the handsearcher identified
84.0 (100/119) of CCTs. Authors may more
readily understand that randomized studies (RCTs)
are trials compared with CCTs. A study is a CCT
if it could possibly be a randomized or
quasi-randomized study, so another explanation is
that some trials are incorrectly classified as
trials when they are not by the handsearcher.
In 2004, authors correctly classified CTs for
229/295 CT abstracts, (sensitivity 77.6).
However, authors failed to recognize 66/295 CTs
(22.4) and mis-classified 405 studies as CTs
that were not (positive predictive value
229/634, 36.1). In 2005, sensitivity was lower
than for 2004 (70.5, 232/329 CT abstracts) and
there were more false negatives (97/329 29.5)
but the positive predictive value was higher
(47.8 (232/485 abstracts)).
COMMENT AND FUTURE DIRECTIONS
METHODS
Classification as Clinical Trial by Handsearcher

• Comment Earlier studies showed handsearching of
  the printed ARVO meeting abstracts requires a
  considerable time investment (about 45 hours per
  annum). Our project to develop a valid and
  reliable system for author classification of CTs
  and electronic searching has the potential to
  shorten this time. Because we found a large
  proportion of abstracts misclassifed (false
  positives) in 2004, the 2005 ARVO abstract
  submission information was modified by changing
  the study design question to ask Is the research
  presented in your abstract a human controlled
  clinical trial?
• Failure to identify abstracts as human CTs
  (false negatives) is of greater concern than
  false positives, as omission is less easily
  addressed than overselection in the
  handsearching process. The proportion of false
  negatives increased, if anything, between 2004
  and 2005.
• Asking authors to classify abstracts identifies
  a small number of abstracts (26 and 22,
  respectively) that would be missed by a
  handsearcher.
• Some abstracts describing CTs may have been
  missed by both the author and the handsearcher
  this is a limitation of the study.
• Future directions
• We plan to explore
• How best to ensure capture of CTs not so
  identified by authors (false negatives).
• Identification of a concise set of text words
  and terms with high sensitivity for
  identification of CTs in electronic-only meeting
  abstracts. These will be validated using
  proceedings from years other than 2004 or 2005.
• Keywords and abstract classification codes
  provided with the ARVO abstracts, to determine
  their usefulness in searching. 

At our request, the ARVO conference organizers
included a question and accompanying definition
on the electronic abstract submission form for
2004 Is the research presented in your
abstract a human clinical trial? Yes No. In
2005, the question was modified to Is the
research presented in your abstract a human
controlled clinical trial? Yes No. For both
years, the definition, obtainable via hypertext
of the blue underlined text, was A planned
study in humans designed to assess the efficacy
and/or safety of one or more test interventions
by comparing outcomes in individuals assigned the
test intervention(s) with those receiving no
intervention or a comparison intervention, and
where individuals in all groups are enrolled,
treated, and followed concurrently. One of us
(RS), masked to authors CT classification,
handsearched 2004 and 2005 ARVO abstracts online.
A gold standard of abstracts classified as
CTs was determined by examination of the two sets
of abstracts 1) abstracts classified as CTs by
the author and 2) those classified as CTs by the
hand-searcher as CTs. Each abstract in the gold
standard set was further categorized as a
randomized controlled trial (RCT) or controlled
clinical trial (CCT), using Cochrane
Collaboration definitions. We evaluated the
author and handsearcher classifications against
the gold standard, and explored factors
associated with author misclassification.
Of the 295 abstracts in the gold standard, the
handsearcher classified 91.1 (269/295) as a CT
in 2004 and 93.3 (307/329) in 2005.
Reasons for Misclassification of Abstracts as a
CT by the Abstract Author
2004
2005
CONCLUSIONS

• More than one reason applied in some abstracts.

The most common reason for misclassification
(false positives) by abstract authors was that
there was no comparison group, ie., the study was
a prospective series with everyone receiving the
same treatment. The second most common reason was
participants were assigned to a group using some
participant characteristic (such as having a
disease or not). Typically, these were case
control studies.

• Neither the author checkox nor handsearching
  identified all CT abstracts.
• Including the checkbox on the abstract
  submission form identifies a small groups of
  abstracts missed by handsearching.
• RCTs were more readily identified as clinical
  trials by both authors or hand- searchers
  compared to CCTs (quasi-randomized or possible
  randomized trials).