Readability Assessment of British Internet Information Resources on Diabetes Mellitus Targeting Layp

1
Readability Assessment of British Internet
Information Resources on Diabetes Mellitus
Targeting Laypersons

• Maged N. Kamel Boulos, Ph.D.
• School for Health, University of Bath
• M.N.K.Boulos_at_bath.ac.uk

2
Agenda

• Introduction
• Material and Methods
• Results
• Discussion and Recommendations
• Conclusions and Useful Web Resources

3
Introduction
4
On Diabetes Mellitus

• Diabetes mellitus, with its different clinical
  types, is an important public health condition in
  the UK and elsewhere, accounting for a sizeable
  percentage of the annual NHS expenditure.
• The education of members of the public, patients
  and their carers is widely recognised as an
  essential component of chronic disease care and
  effective health promotion.
• Appropriate education can minimise the risk of
  diabetes mellitus and its complications.
• The Internet could be an extremely useful and
  cost-effective medium in this respect.

5
On Readability

• Readability is an attempt to match the reading
  level of written material to the reading with
  understanding level of the reader.
• Readability (or understandability) is a
  frequently overlooked aspect of health
  information quality and accessibility.
• For example, a recent evaluation by Thakurdesai
  et al. (2004) of the quality and contents of
  diabetes mellitus patient education on the
  Internet failed to test the reviewed sites for
  readability.

6
On Readability Contd

• Nearly half of all American adults or 90 million
  people have limited English and math skills (most
  are native-born English speakers), making it
  difficult for them to fully and safely understand
  and act upon health information (US Institute of
  Medicine, 2004).
• Comparable proportions have been reported in the
  UK according to the Office for National
  Statistics.

IOM Report Cover
7
Readability Testing Methods

• More costly and time-consuming, but also more
  precise methods
• A comprehension test based on the document of
  interest administered to a group of readers of
  known reading ability.
• Measuring readability by the judgment of a
  literacy expert.
• Quick and rough approximations
• Using readability formulae mathematical
  equations that estimate the reading level of a
  document based on the words that are used and the
  lengths of sentences.
• Dozens of different readability formulae (e.g.,
  Fry, SMOG and Flesch/Flesch-Kincaid formulae)
  have been developed over the years and many are
  implemented today in software.

8
Material and Methods
9
Sample Selection

• We used Google (http//www.google.co.uk/ - UK
  pages) to locate British Internet information
  resources on diabetes mellitus targeting
  laypersons.
• The links sections of some of these resources
  were also explored to pick further UK sites.
• In total, we selected 15 trustworthy English
  language sites/providers for inclusion in this
  study (all of UK provenance).

10
Readability Formulae and Software Used

• We tested the readability of 20 representative
  pages from these 15 sites, providing consumer
  information on diabetes mellitus.
• The full text from each of these pages was opened
  in Microsoft Word 2000 (any page navigation text
  was removed), and a readability level was
  generated using the Flesch Reading Ease score and
  Flesch-Kincaid Grade Level score.

11
Readability Statistics in MS Word
Readability statistics in Microsoft Word for the
text of http//www.diabetes.org.uk/infocentre/pubs
/Whatc.doc The dialogue box on the left shows
where to enable this function (Show readability
statistics) in Microsoft Word.
12
Calculation of Flesch Reading Ease and
Flesch-Kincaid Grade Level scores
13
Transforming Results to Equivalent UK Reading Age
and Computing Statistics

• The reading level in US school grade
  (Flesch-Kincaid) was then transformed to the
  equivalent UK reading age.
• For example, a Flesch-Kincaid Grade Level score
  of 8.0 equates to the UK reading age of an
  educated 13 year old person.
• Mean and median readability scores of all
  sites/pages and other statistics were computed in
  Microsoft Excel 2000.

14
Results
15
Required Reading Age of Tested Material

• The average and median UK reading ages of the
  sites/pages tested were 14.2 and 14.65 years of
  age respectively (range 11-16.9 years).
• These figures are well above the estimated
  reading age of the UK population in general (9
  years or Year 4/5 at UK school).

16
Results Contd

• Only 7 pages (out of 20 tested, i.e., 35 of all
  tested pages) from 4 sites/providers (out of 15
  included in this study, i.e., 26.66 of all
  sites/providers evaluated in this study) had a
  Flesch Reading Ease score above 60 (range
  60.4-73.3) and a Flesch-Kincaid Grade Level of
  8.0 or lower (range 6.0-8.0) corresponding to a
  UK reading age of 11-13 years.
• The remaining sites/pages in this study scored
  worse (Flesch Reading Ease score range 31.6-57.1
  and Flesch-Kincaid Grade Level range 9.2-11.9),
  with a corresponding UK reading age of 14.2-16.9
  years.

17
Detailed Results
18
And the Winner Is

• NHS Direct Online health encyclopaedia
  (http//www.nhsdirect.nhs.uk/) scored among the
  worst sites in this study, while the patients
  section of BestTreatments (BMJ Publishing
  Group/NHS Direct - http//www.besttreatments.co.uk
  /) came out as the best, and most readable site
  in our study.
• As expected, the two readability scoring formulae
  used in our study were found to be very highly
  correlated (a high negative correlation of
  -0.92124, measured using Microsoft Excel CORREL
  function when Flesch Reading Ease increases,
  Flesch-Kincaid Grade Level decreases and vice
  versa).

19
Discussion and Recommendations
20
How Does This Study Compare With Previous Studies?

• Our results echo those of similar studies
  conducted over the past few years, and point to a
  chronic and often forgotten information quality
  problem.
• Berland et al. (2001) reviewed 25 English and
  Spanish-language health Web sites and concluded
  that much of the health information available on
  the Internet is beyond the comprehension of many
  consumers.
• Kusec et al. (2003) studied the readability
  levels of 99 diabetes-related Web sites
  displaying the HONcode logo (http//www.hon.ch/)
  and found that 86.9 of the tested material would
  be too difficult to read for the average adult
  population.
• Chestnutt (2004) evaluated the readability of 100
  patient-related Internet sites presenting
  information on common oral pathologies and
  reported a mean UK reading age of the evaluated
  sites of 10-11 years, which is higher than the UK
  national reading age.

21
Limitations of Readability Formulae

• There are many factors that affect readability
  and understandability of online health
  information, which are not measured by
  readability formulae.
• Rudolf Franz Flesch, who created the Flesch
  readability formula, warned that Some readers, I
  am afraid, will expect a magic formula for good
  writing and will be disappointed with my simple
  yardstick. Others, with a passion for accuracy,
  will wallow in the little rules and computations
  but lose sight of the principles of plain
  English. What I hope for are readers who won't
  take the formula too seriously and won't expect
  from it more than a rough estimate.

22
But They Remain Better Than Nothing...

• Nevertheless, we recommend that any health
  information quality benchmarking scheme or
  checklist in use should include (or be
  supplemented with) some explicit testing for
  content readability.
• Readability testing results could be also
  displayed on the tested pages to inform readers
  (a readability seal).

23
Making Online Consumer Health Information Easier
to Understand

• Ensure that written material is pitched at a
  level appropriate to the intended lay audience
  (aim to match the general reading level of the UK
  population nine years or Year 4/5 at UK school).
• Write shorter sentences and paragraphs.
• Use the active voice.
• Avoid jargon and use simpler, common words.
• A picture is worth thousand words include real
  world examples and easy-to-understand
  illustrations (where applicable).

24
Practical Example (Re-)writing for Readability

• Quoted fromhttp//www.pfizerhealthliteracy.com/im
  proving_empathy.html
• Original text - Flesch Reading Ease 14.5 A
  living will is a written declaration directing
  your doctor to provide, withhold or withdraw
  lifesaving procedures should you be diagnosed as
  having a terminal condition or exist in a
  vegetative state and you lack the capacity to
  make the decision. (Source Planning for
  Incapacity, Legal Council for the Elderly)
• Revised text in an easy-to-read format - Flesch
  Reading Ease 68.1 ( better) A living will is
  a paper you sign telling your doctor whether you
  want treatments that could keep you alive if you
  have a terminal disease or are in a coma. This
  lets your doctor know how you want to be treated
  if you are not able to make the decision yourself.

25
Making Online Consumer Health Information Easier
to Understand Contd

• Properly structure and organise information into
  sections and subsections with meaningful headings
  and subheadings.
• Use bulleted lists (where applicable).
• An inverted pyramid writing style is sometimes
  also helpful (start the article by telling the
  reader the conclusion/action(s) to take, followed
  by the most important supporting information, and
  end by giving the background).

26
Making Online Consumer Health Information Easier
to Understand Contd

• Consider the specific cultural needs and
  socio-economic levels of different ethnic groups.
  For example, health education material should
  describe alternative options in case of lack of
  economic resources, when referring to processes
  that imply large expenses, e.g., some costly
  diabetic diet recipes.
• Also pay attention to the visual style and
  presentation of online health information
  material, including issues like text/background
  colour and font type and size. These are equally
  important to improve information accessibility.

27
Consider Other Means of Delivery

• However, even the most readable documents will
  remain difficult to fully and properly understand
  for a sizeable proportion of the population.
• For this reason, in addition to written online
  material, health information providers should
  also consider other means of delivering health
  education and promotion programmes like
  face-to-face education and plain English videos,
  so that no one is left behind.
• The videos could be also made available online
  and provided in additional languages spoken by
  ethnic minorities/immigrants.

28
Conclusions and Useful Web Resources
29
Not All Patients Are Experts!

• During recent years, there has been a growing
  trend of treating medical knowledge as a single
  blob that is relevant to both professionals and
  laypersons, and so should be made accessible to
  all groups without any distinction.
• Supporters of this trend believe that patients
  should be empowered and given more information
  and control of their conditions. Indeed,
  laypersons sometimes show more knowledge and
  understanding of their own conditions than their
  treating doctors do.
• In spite of that, it seems sensible to continue
  writing different information sets for the
  healthcare professional, the highly educated and
  the average layperson, and to label online health
  information according to its intended primary
  audience.

30
Conclusions Contd

• Besides the fundamental requirement to check the
  medical correctness or soundness, completeness,
  accuracy, and currency of health information
  targeting laypersons, online health information
  providers also need to ensure that their written
  material is pitched at a level appropriate to
  their intended lay audience.
• This is an equally important requirement as it
  affects the final utility and reliability of
  published online material.
• Public and patient health information that is
  difficult to understand or liable to
  misunderstanding by the lay consumer could result
  in serious consequences.

31
Useful Web Resources

• Pfizer Clear Health Communication -
  http//www.pfizerhealthliteracy.com/and their
  handbook - http//www.pfizerhealthliteracy.com/454
  8_Health_literacy_all.pdf
• The Center for Health Care Strategies Fact
  Sheets on Health Literacy - http//www.chcs.org/