Quality Metrics for IRBs Annette R. Anderson, MS, RHIA, CIP

1
Quality Metrics for IRBs

• Annette R. Anderson, MS, RHIA, CIP
• VA Central IRB Administrator
• Program for Research Integrity Development and
  Education (PRIDE)

2
Overview

• What are quality metrics and how can they be
  used?
• What are the current problems in using these
  concepts to measure the quality and effectiveness
  of IRBs?
• How can we use metrics in our local programs to
  measure and improve the performance of our IRB?
• What kind of metrics should be used in the future
  to measure the quality and effectiveness of IRBs
  and/or are different types of tools needed?

3
Background

• Quality and Metrics definitions differ
  depending on upon context, industry, and
  viewpoint
• Used most in business, engineering, and
  manufacturing
• Adopted by many other sectors (e.g., the
  government and health care industry)
• Applying the business principles underlying
  Quality Metrics to IRBs can be problematic

4
What is the Meaning of Quality

• Definitions
• An inherent or distinguishing characteristic a
  property or attribute
• A personal trait, especially a character trait
• Essential character nature
• Superiority of kind
• Degree or grade of excellence
• ISO 9000 Degree to which a set of inherent
  characteristics fulfills requirements
• A perceptual, conditional, and somewhat
  subjective attribute understood differently by
  different people

5
Two Common Quality-Related Functions

• Quality Control - the process of reviewing the
  quality of all factors involved in the production
  of goods or services
• Emphasizes testing and verification
• Use of audits and checklists to uncover defects
  or deficiencies
• Quality Assurance systematic monitoring and
  evaluation of the various aspects of a process,
  service, or facility to maximize the probability
  that minimum standards of quality are being met
• Emphasizes validation
• Uses metrics to prevent deficiencies and measure
  progress toward set goals

6
Definition of Metrics

• Verifiable measures stated in either quantitative
  or qualitative terms that
• Capture performance in terms of how something is
  being done relative to a standard or target
• Allow and encourage comparison
• Support the goals and objectives of the
  organization
• Performance metrics (i.e., Key Performance
  Indicators) often represent how far a metric is
  above or below a specified target

7
Use of Quality Metrics

• Effective quality metrics must
• Have a support system to perform ongoing
  monitoring
• Be used to inform decision making
• Focused on seeking corrective actions that build
  a foundation for continuous process improvement
  just collecting and reporting the data is not
  enough
• Quality metrics can be used to
• Spot trends in performance (good and bad)
• Adjust processes per institutional goals and
  objectives
• Compare alternative processes
• Compare to both inside and outside benchmarks
  (best practices)
• Predict performance

8
Quality Metrics Applied to IRBs - Do IRBs Protect
Human Subjects?

• Researchers, Institutions, IRB members, and IRB
  staff have all complained about IRB mission
  creep, excessive paperwork, and attention to
  inconsequential details
• Studies have focused on structure, administrative
  efficiency, and procedural compliance with
  federal regulations and local policies
• Data suggest a need for more efficiency and less
  variation among IRBs
• Little is known about whether or not IRBs achieve
  their goal of enhancing human subject protections
  in research

9
Other Problems in Developing and Using Quality
IRB Metrics

• No agreed upon metrics or outcome measures for
  evaluating IRB effectiveness
• Each Institution often has its own way of
  calculating how a metric is defined and measured
  (e.g., turnaround time)
• Practices and decisions vary from IRB to IRB and
  even within the same IRB
• Who is the customer (e.g., Institution,
  investigators, or human subjects?)

10
No Generic One Size Fits All IRB Metrics
Organizational Factors to Consider

• Commitment from facility and research leadership
  to
• Allocate resources for collection and analysis of
  metrics data
• Use the data in their decision making
• Communicate and coordinate with all stakeholders
  transparency is key
• Fit organizational culture, goals and objectives
• Take into account Institutions existing
  structure
• Size and complexity of the Institution and
  research program
• Type of IRB arrangement
• Available resources

11
Additional Factors to Consider

• Top down drivers what is most crucial to the
  achievement of your organizational goals and
  objectives?
• Set attainable targets and/or select benchmarks
  (ensure they are compatible) or establish
  internal comparisons
• Are you already collecting the metric in some
  form or will you need to implement a new process?
• Balanced Approach
• How many metrics are too much or too little
• How often should a particular metric be collected
• How often should it be analyzed and/or reported?

12
Data Analysis

• Crucial for monitoring progress and identifying
  trends
• Tell what is going on and what is needed to
  improve
• Justify or explain outliers
• Present data with and without outliers
• Present mean, median, and mode
• Gather additional data if needed to better
  diagnose and resolve problem areas
• Prepare and share draft reports to get additional
  input

13
Take Appropriate Action

• Authority to take action should be at lowest
  level possible
• Change policy or procedures
• Add additional resources (increase staff/use of
  technology)
• Obtain additional training or consultant services
• Re-measure and perform ongoing monitoring to
  ensure problem resolved and/or does not recur
• Publicize results and emphasize any improvements
  or areas that need attention
• Adjust targets based on performance are they too
  low?
• Re-evaluate appropriateness and effectiveness of
  metrics should be fluid, not static

14
IRB Metrics Examples Turnaround Time

• Must define what this means and set target(s)
• Submission date to final approval date or
  approval contingent on minor modifications?
• Includes investigator response time?
• When does clock start?
• Does clock start again if investigator does not
  respond within defined timeframe?
• Separate measurements at different process
  points?
• Separate measures for expedited and exempt
  reviews?
• How measured? (e.g., business or calendar days?)

15
IRB Metrics Examples IRB Staff Workload and
Performance

• Define how workload will be measured and set
  target
• Cradle to grave system or functional system?
• Factors to consider
• Number of studies overseen per staff member by
  type of study
• Number of actions overseen by staff member by
  type of action
• Complexity, risk, and duration of the study
• Timeliness and completeness of administrative
  pre-reviews and drafting of determination letters
• Other duties assigned that have impact on ratio
• Training and experience (i.e., of CIPs
  training opportunities given per time frame)
• Turnover metric

16
IRB Metrics Examples IRB Member Workload and
Performance

• Quantitative Factors to Consider
• IRB size and member qualifications
• Number of studies per reviewer by study type
• Number of actions reviewed by IRB member by type
  of action
• Complexity, risk, and duration of the study
• Timeliness of Reviews
• Attendance at meetings and training events
• Frequency of use of Alternate Reviewers and
  Consultants

17
IRB Metrics Examples IRB Member Workload and
Performance (Cont.)

• Qualitative Factors to Consider
• Depth and thoroughness of reviews
• Adequate completion of checklists
• Logical and coherent motions
• Give individual feedback

18
IRB Metrics Examples IRB Meeting Management

• Factors to Consider
• Number and type of IRBs local or affiliate
• Frequency and length of meetings
• Number of items on agenda
• How often is quorum lost and why?
• How often items tabled to next meeting and why?
• Did all members receive all required materials in
  time
• Minutes
• How soon after the meeting are minutes available
  for review?
• How many corrections must be made to minutes?
• Are all expedited and exempt actions
  appropriately reported?

19
Other IRB Metrics

• Attaining and maintaining accreditation
• Monitoring of audit reports to ensure compliance
  with regulatory requirements
• Internal monitoring process for review of IRB
  records
• Number of complaints (i.e., investigator,
  participants)
• Quality and completeness of IRB communications
• Feedback from
• IRB Members
• IRB Staff
• Investigators
• Other organizational stakeholders

20
A Little Bit of Quality Metrics Philosophy

• From a presentation by Dr. Stephen Rosenfeld,
  former President and CEO of Western Institutional
  Review Board, given on June 21, 2010
• Metrics rarely tell us something we dont already
  know, but when they do, its important
• They force us to make our assumptions explicit,
  and sometimes force us to abandon our assumptions

21
Quality Metrics Appear to Be Here to Stay

• Decision making is increasingly data driven
• Successful organizations are shifting to
  evidence-based management
• Data is used to back-up decisions as a way to
  manage risk

22
The Future for IRBs A Back to Basics Movement

• Research has changed dramatically since IRBs were
  created
• Current metrics on IRB workload continue to be
  important however, they do not necessarily
  measure the effectiveness of the IRB in enhancing
  human subjects protections
• Growth of centralized IRBs
• Symposia, conferences, and papers on alternative
  models

23
The Future of IRB Quality Metrics

• Movement to base metrics on IRB approval criteria
• Challenge Converting these concepts into valid,
  reliable, and verifiable metrics
• The Clinical Research Group of the NIH-funded
  Clinical and Translational Science Awards
  Consortium has begun an initiative delineating
  measures of IRB quality

24
Recommended Reading

• JAMA Commentary, September 8, 2010 Vol 304, No.
  10, by Dr. Christine Grady, Department of
  Bioethics, National Institutes of Health Clinical
  Center, Bethesda, Maryland
• Holly A. Taylor, Moving Beyond Compliance
  Measuring Ethical Quality to Enhance the
  Oversight of Human Subjects Research, IRB
  Ethics Human Research 29, no. 5

25
Questions and Contact Information

• Annette Anderson, MS, RHIA, CIP
• VA Central IRB Administrator
• 202-461-1813 annette.anderson3_at_va.gov
• Website
• http//www.research.va.gov/programs/pride/cirb/def
  ault.cfm
• Toll free number 1-877-254-3130
• Email va.central.irb_at_va.gov