Cleft Speech Audit Studies in the United Kingdom: Achievements, Challenges and Future Directions

1
Cleft Speech Audit Studies in the United Kingdom
Achievements, Challenges and Future Directions

• Debbie Sell, Alex John, Anne Harding-Bell, Triona
  Sweeney, Fran Hegarty, Jenny Freeman, Sue
  Mildinhall, Sarah Collins

2
Structure to the Presentation

• Part 1 Development of the tool
• Part 2 Training Programme
• Part 3 An example of a national inter-centre
  audit study

3
Part 1 Development of the Tool
1998 UK CSAG Report on Cleft Lip and Palate
Report recommended Systematic regular
monitoring of outcomes in cleft palate is
mandatory - intra-centre and inter-centre audit
4
Perceptual Speech Assessment

• No one accepted approach
• Debate on speech sample content
• Debate on audio versus video
• Existing rating scales ill-defined
• Original tool (CAPS Cleft Audit Protocol for
  Speech) not tested for validity and
  reproducibility

5
National UK Audit Tool Cleft Audit Protocol for
Speech Augmented - CAPS-A
The development of a valid, reliable and
applicable audit outcome tool John, Sell,
Sweeney, Harding-Bell and Williams, 2006, CPCJ
43, 272-288.
6
Method

• Stage1
• Defining the scale -
• 3 specialist SLTs to act as expert panel to
  decide on content and format of the audit tool.
• Stage 2
• Testing the scale -
• Pilot 1 - 7 specialist SLTs - audit experience
• Pilot 2 - 7 different specialist SLTs - audit
  experience
• Validity - face, content, criterion validity
• Reliability - test-retest inter/intra-rater
  reliability
• Acceptability - qualitative feedback from users

7
Cleft Audit Protocol for Speech-Augmented
8

CAPS- A (CAPS Augmented)
9
CAPS A Hypernasality Scale
10
Pilot 1

• Results -
• Criterion Validity - good
• Reliability - intra inter-rater
• Variability across the sections of the tool on
  complex cases
• Action -
• Expanded accompanying definitions
• Refined rating points
• Training points identified

11
CAPS Standardised Reporting of Outcomes
according to Colour

Harland, 1996
12
Inter-Rater Reliability
Inter class correlation coefficient
13
Articulation Errors
Posterior Errors
Anterior Errors
Non Oral Errors
and
(Harding et al, 1997)
Passive Errors
14
Inter-Rater Reliability
Inter class correlation coefficient
15
Pilot 2 Intra-Rater Reliability
16
Acceptability

• Questionnaire data
• Reported as acceptable easy to use with
  training

17
Part 2 Perceptual Training Programme
Cleft Audit Protocol for Speech (CAPS-A) A
Training Package for Speech and Language
Therapists Sell D. John A. Harding-Bell A.
Sweeney T. Hegarty F. Freeman J. IJLDC 2009
18
Progress to date

• 6 courses to date/74 therapists trained in total
• Intra and inter-rater reliability information
  provided
• 2009 4 more trainers being trained ie Training
  the Trainers

19
Method

• 1.5 days CAPS-A training on definitions and
  listening
• 0.5 day Consensus listening
• Day 3 Independent Listening of 10 cases
• Day 4 Repeat of day 3 at least one month later
• 2009 Baseline listen added

20
Training Package included

• CAPS-A forms/definitions
• Revised picture book
• Phonetic Cribsheet
• Training notes
• Intra-rater and
• inter-rater reliability results

21
Standardisation of Process Issues

• Speech sample
• Method of elicitation
• Recording and playback equipment and guidelines
  background, lighting, picture composition,
  microphone
• Structured Listening Protocol (Kent 1996)
• Very similar to www.eurocran.org
• Recommendations for recording and playback
  techniques reflect standards in the broadcast and
  telecommunications industry

22
(No Transcript)
23
Results Combining Results across 2 Occasions

• Generalisability coefficient
• measures extent to which it is possible to
  generalize between individuals and patients
  averaging across occasions increases sample size
  and more precise inter-rater reliability measure
• Test-retest coefficient
• measures test stability over time

24

Generalisability and Test-Retest N 36
Strength of Agreement
Strength of Agreement
0.60
0.67
Mod
Good

ICC Intraclass correlation coefficient
25
Generalisability and Test-Retest N 36
Strength of Agreement
Strength of Agreement
Generalisability
Test-Retest
26
Conclusion

• Training in the main effective
• Uncertainty about level of reliability that can
  be expected for some parameters
• Standardisation of process issues

27
Part 3 An example of a national inter-centre
audit study
Piloting UK Inter-centre Speech Audit Is it
possible? Sue Mildinhall, Debbie Sell,Sarah
Collins Paper presented at the meeting of the
Craniofacial Society of GB and Ireland, 2007
28
The Audit Cycle
5.Implement changes
2.Set/identify standards
1.Identify Issue
3.Data collection and analysis
4.Compare performance with standards
Courtesy of Tri-Centre Audit Group
29
Aims of the Project

• To pilot undertaking a collaborative national
  inter-centre speech audit of children of 5 years
  old with UCLP
• To compare results in 2000 against UK 1995 CSAG
  national study
• To compare results against a defined standard
• To pilot the Speech Therapy History Forms

30
Method

• Each Regional Centre identified 10 children with
  UCLP born from 1st January 2000
• Consecutive series provided with exclusions with
  reasons
• Surgical and speech therapy history gathered
• Surgery by 27 different surgeons

31
Subjects

• 127 patients identified
• 35 exclusions
• 11 No audit carried out
• 5 Failed to attend audit session
• 6 Unable to complete audit task
• 7 Recording error
• 3 Had moved away
• 92 remained
• 63 (n58) male 37 (n34) female
• All patients had standardised speech recordings
  using the same speech sample

32
Method cont.

• One centre declined to take part
• All listeners trained on CAPS-A (John et al,
  2006)
• 5 Listening groups
• Independent consensus listening
• (Sell et al, 2009)

33
Result 1 Piloting undertaking an
inter-centre speech audit

• 11/12 centres willingly participated
• All teams reached deadlines efficiently and
  effectively
• Listening process successful and beneficial for
  CPD
• National inter-centre audit possible

34
Result 2 Speech Errors Births 2000 v CSAG Study
Births 1991-1992

Developmental errors 63
35
Resonance Births 2000 v CSAG Study Births
1991-1992

36
Colour coding defines outcome

37
Result 3 Draft Speech and Language Therapy
Standards (Britton et al, unpublished)

• Standard By 5 5.11 years 70 of children will
  have speech within the normal range green
  profiles on CAPS-A (CSAG 1998, RCSLT Clinical
  Guidelines, Kuehn and Moller 2000)
• Result 37 normal
• 10 of non cleft 5 year olds have abnormal speech
  (McLeod 2002)
• Standard needs revision

38
Result 4 To trial Speech Therapy History Forms

• 71 (n65) Data returned
• 71 (n46) Had received therapy
• 36 gt 20 sessions SLT ie major speech problems?
• Difficulty interpreting methods of delivery of
  therapy, reasons for lack of progress
• Inadequate data regarding attendance
• Inadequate information on quantity/quality of
  therapy

Difficult to get retrospective meaningful
detailed data
39
Conclusion

• Tool and training programme devised and
  implemented
• National inter-centre speech audit possible
• Excellent CPD opportunity
• Small number of cases analysed per day
• Outcomes similar to UK 1995 CSAG study but still
  multiple surgeons in 2000
• Standards for audit under current review
• Retrospective therapy history basic data only

40
Acknowledgements

• Lead Speech/Language Therapists Christine
  Hayden, Melanie Bowden, Siobhan McMahon, Jan
  Wilson, Liz Albery, Ginette Phippen, Lorraine
  Britton, Jane Russell
• All UK Cleft Speech/Language Therapists
• All Regional Teams for financial support for the
  training programme
• CRANE for financial support for CAPS-A development