Themes that Cut Across Disciplines: Quality Control and Research Methods

1
Themes that Cut Across Disciplines Quality
Control and Research Methods

• Faina Linkov, PhD
• Research Assistant Professor
• University of Pittsburgh Cancer Institute

The journey of quality control in the
Supercourse The journey of a thousand miles
starts with a single step
2
Overview

• Overview the history of quality control and peer
  review
• Quality control on Amazon.com
• Quality control in the Supercourse QC on PPT
  lectures is at the core of what we do
• Supercourse studies of quality control
• Our conclusions and future directions

3
What do scientists have in common?

• Quest for discovery
• Common research methods, such as statistics
• Journal article format
• Quality control mechanisms
• Peer review is it the same as quality?

4
History of Quality Early times

• 384-322 BCE-Aristotle and the birth of peer
  review
• 854-931 Ethics of the Physician by Ishap bin
  Ali Al Rahwi-first documented description of peer
  review in Medicine

5
Royal Society of London

• Philosophical Transactions- Originally edited by
  Henry Oldenburg

First peer reviewed journal
1665
6
Typewriter the tool that fueled peer review
process
1890s
7
Quality control criteria of the past

• Albert Einstein, 1905

His famous work, Annus Mirabilis, was not peer
reviewed!
8
History of quality continued
19th century birth of quality control
discipline 1950 - W. Edwards Deming and
statistical quality control 1970s- Total
Quality Management (TQM)
9
What is Quality

• Doing the right thing right, right away
• W. Edwards Deming, 1982
• a measure of goodness that relates to the
  intended use of a product and the expectations
  customers have concerning this product
• 
  Barkman, 1989

10
Traditional Quality control of Scientific
Materials Peer Review

• A peer-reviewed journal is one that has
  submitted most of its published articles for
  review by experts who are not part of the
  editorial staff
• Vancouver Group of
  Editors

11
Is peer review the same as quality?
South Korean cloning scandal, alongside with
other examples, demonstrate that peer review is
not perfect
12
Peer review congress finding new ways or fixing
the old system?
13
The State of QC for PowerPoint on the Web

• Over 7,600,000 PowerPoint files on the Internet
  give us unique new ways to do QC
• Only 3383 lectures tested for QC-Supercourse
• We published almost all articles concerning the
  state QC of PowerPoint on the web

14
Amazon.com the first website that allowed to
post a book review
15
PowerPoint lectures

• No information on quality control for PowerPoint
  lectures on-line
• There is a growing need to establish quality
  control for Scientific PPT lectures on Line
• Migration of 7 million lectures to the web by
  2008
• Establishment of Quality Circles in the
  Supercourse John Last, Vint Cerf, and many others

16
Progress Report in the Supercourse

• Over 14000 lecture reviews were obtained in the
  last 6 years

Lecture review forms utilize Likert like scales,
regularly used for the evaluation of educational
programs and amazon, ebay, consumers reports..
17
Space for written comments
Some of the URLs have changed since the lecture
was developed. It would be helpful if that was
corrected, so you could actually see the
information referred to. Is there any text(s)
that you could recommend for epidemiology, i.e.,
a basic text, an intermediate text, and an
advanced text?
18
My dissertation, 2005

• First evaluation of peer review mechanisms for
  PPT lectures on-line

19
7 expert reviewers from 6 countries
20
Results and conclusions of this research

• Lectures were rated highly by experts and
  non-experts
• People are primarily rating at the upper tail of
  the distribution
• There is a poor agreement among reviewers (very
  low Kappa statistic)
• The role of personal background in the quality of
  the peer review is not clear and needs to be
  further investigated
• There is a pressing need for scientific
  evaluation of PPT lectures on the web

21
Implications for the Supercourse of Science?

• There is no consistency about the use of peer
  review on the web
• Utility of peer review strategies for PPT
  lectures is not clear
• New approaches will be developed for the
  Supercourse of Science.
• Shall we allow disciplines to decide the QC
  techniques which are most applicable for their
  fields?
• There is a need to build a science of QC

22
Where do we go from here for the
development of quality control mechanisms in the
Supercourse?
Continuous quality improvement (CQI) not a
destination, but a journey
23
Supercourse aim

• To scientifically develop a system of QC on the
  web by making multiple measures available
• Power of the Internet makes the development of
  the Science of QC possible.

24
Moving beyond peer review

• We had great difficulty in finding any real
  hard evidence of the systems effectiveness,
  which is disappointing, as peer review is the
  cornerstone of editorial policies worldwide
• 
  Tom Jeffersen

25
Suggested Quality Control Mechanisms for the
Supercourse of Science Multiple Measures

• a) Expert editorial board
• b) Screening to identify non-appropriate
  lectures, as done in nature and BMJ
• c) Five star system (or Priority Score) similar
  to that in Amazon.com
• d) Opinion of experts (professors)
• e) Personal Characteristics of authors (rank,
  university, citations)
• f) Web statistics for utilization of lectures
  hits, links, Page Rankings, etc.
• g) Key note speeches
• h) Publications and citations from Google Scholar
• i) Model similar to NIH style review

26
Editorial Board (in formation)

• John Last, MD
• father of modern environmental epidemiology
• Peter Greenwald, MD, PhD
• - leading cancer control specialist, NCI
• Paul Zimmet, MD
• -leading diabetes research in Australia
• Dr. Steven N. Blair
• -leader in exercise science
• Plus 8 others

27
Screening to eliminate poor quality lecture

• Key criteria for lecture rejection in the parent
  Supercourse project
• Not related to prevention
• Inaccurate
• Out of date
• Incomplete

28
Amazon.com like system
Customer Reviews ltBook titlegt
42 Reviews
29
Mean Lecture Scores in the Supercourse

• First 2000 surveys
• Mean Score 4.12

30
Personal Characteristics of Authors(rank,
university, citations)

• 1288 authors in the Supercourse
• 30 authors from Harvard
• 8 authors from Johns Hopkins
• 100 authors from Pitt
• 7 authors from Columbia

31
Web statistics
32
Keynote speeches
Someone chosen for keynote address is expected to
give a good lecture
33
NIH Model and Continuous Quality Monitoring
Priority Scores
Triage
Resubmission
Submission
Supercourse Model Continuous Quality Control
34
Google model for QC Opportunities for
exploration?

• Do not exclude anything, but give it a rank
• Development of search mechanisms where you can
  find anything you like

35
Why do this research?

• I believe this lecture provides simple,
  understandable, thorough and consolidated
  information of the subject. This is actual
  service to the humanity
• From the evaluator of the
• Supercourse lectures

36
Conclusions

• There is a need to define the science of QC for
  online lectures
• Scientists need to be able to choose what they
  want in terms of QC mechanisms, including peer
  review and other systems
• Science needs to replicate success of industry in
  the area of QC
• Traditional approaches (such as peer review) may
  not be bad, but with Internet technologies much
  more can be done

37
Vision for the Supercourse of Science

• Development of novel QC systems for the
  Supercourse of science.
• Need to systematically and continuously evaluate
  QC systems
• Upfront QC systems and information collection

38
Thank you!

• Please e-mail your ideas to super1_at_pitt.edu