A Reusable Template for Evaluating Point-of-Care Information Products

1
A Reusable Template for Evaluating Point-of-Care
Information Products
2
Discussion of EBM Products on MEDLIB-L
How many questions have there been about EBM
resources in recent years?
How often were specific products mentioned?
(Jan-Nov 2003)
3
Characteristics of Point-of-Care (POC) Products

• Provide access to succinct evidence-based
  diagnosis and treatment information supporting
  care of individual patients
• Oriented to
• primary care physician
• specialist seeking information outside his/her
  own specialty

4
Characteristics of POC ProductsThe Core Document

• Core document is a topic review presented in a
  product-specific, standardized structure.
• Core document is evidence-based.
• Core document content is subdivided for quicker
  access.
• In addition to the core document, product may
  incorporate other resources (journals, textbooks,
  drug information, news or alerts, calculators).

5
Steps in This Study
6
POC Products in this Study

• Product selection criteria
• Ostensibly, all have characteristics listed
  above.
• Product or vendor known directly or by reputation
• Willing to be evaluated or tested against other
  products
• Products evaluated
• Micromedexs DISEASEDEX
• InfoRetriever
• Ovids Clineguide (since subsumed into Skolar)
• MD Consults PDxMD (since renamed FirstConsult)

7
Development of Evaluation Template

• Topic sources
• Informal brainstorming among project librarians
• Brief exposure to some of the products
• Comparison with known products
• Format
• Questionnaire
• Structured and hierarchical
• Reusable in further or additional product
  evaluations

8
Template Topic Areas

• Content
• Scope
• Methodology and core document structure
• Editorial practices
• Architecture/Navigation
• Content storage and organization
• User experience
• Integration with other environments
• PDAs
• Local clinical information systems
• Niche (self-perceived product strength)

9
Interviews

• Questionnaire was used in scheduled vendor
  interviews (3 interviews in person, 1 by phone).
• Group situation 1-2 product developers or
  representatives were interviewed by 2-3
  librarians.
• One vendor followed up by e-mail with additional
  information.
• Product/attribute comparison grid was created in
  Excel and populated as interviews were completed.

10
Literature Review

• Challenges identified
• Information needs
• Information-seeking behaviors
• Barriers to implementation
• Solutions proposed
• Evidence Cart (Sackett Strauss, 1998)
• Remaining questions
• Impact on clinical practice
• Product comparison

11
Wish List Based on Literature Review What would
the ideal product be like?

• Broad scope
• Help with queries and search strategies
• Rx recommendations
• Drug information
• Practice guidelines with automatic EBM updates
• Synthesis of evidence
• Patient education modules

• On demand at point of care
• Linked directly to relevant literature
• Customized for local use
• Flexible decision-making models
• Fast
• PDA compatible
• Conformity to hardware and software standards

12
Product Comparison Scoring Instrument

• Scores for 6 categories (4 in original
  questionnaire)
• Content
• Audience
• Integration
• Architecture/Navigation
• Retrieval (originally under Content)
• Quality Control (originally under
  Architecture/Navigation)
• Scores weighted to favor Wish List compliance and
  
• non-proprietary standards
• Scoring instrument can be tailored to local
  preferences.
• Different weights ? different scoring outcomes

13
Scoring Scheme

• Attributes were listed for each category.
• For every product, each attribute was assigned a
  score.
• Default 1 if attribute present, 0 if not present
• Scores weighted for attributes of particular
  interest
• 2 points for presence of each Wish List attribute
• 0 points for presence of an attribute if it
  involved a proprietary standard
• Points for scores in all categories were totaled.

14
ExampleScoring Instrumentfor Attributes
Associated with Content
15
ExampleOne Scoring Outcome (Total Points)
16
ExampleOne Scoring Outcome (Points by Category)
17
Limitations of This Work

• Interview data were varied and contained gaps.
• There was either no or limited hands-on use of
  products themselves.
• Outcome depends on scoring scheme ours may not
  be optimal for other researchers or libraries.
• This methodological approach disregards real
  world constraints such as
• Product cost
• IP restrictions and other implementation issues

18
Recap

1. Experience with familiar products can be used to
   generate a structured, hierarchical template for
   use in product evaluation.
2. Requirements for point-of-care products can be
   distilled from a literature review.
3. Product characteristics can be scored using a
   locally weighted instrument.
4. Template can be adapted and reused with
   additional products or for further evaluation
   based on product testing.