Outcomes of Social Work Education OSWE: Findings from the Projects

1
Outcomes of Social Work Education (OSWE)
Findings from the Projects

• Hilary Burgess, John Carpenter, Joanna Fox,
  Roxana Anghel, Sharon Vitali, Juliet Koprowska,
  Kish Bhatti-Sinclair, Anne Quinney, Imogen
  Taylor, Clare Ockwell, Suzy Braye, Michelle
  Lefevre

2
Outline of Session

• Introduction to OSWE project
• Presentation of findings from six partners in
  England with questions for clarification.
• Discussion of the findings in general and their
  implications.
• Key issues in doing outcomes-focussed research on
  Social Work Education.
• Open discussion.

3
Aims of OSWE (3 year project)

• To test the feasibility of outcome measures and
  research designs in Social Work Education
• To generate high quality evidence about the
  effectiveness of methods of SWE
• To build capacity and capability amongst
  academics, including service users
• To use opportunities to compare and contrast
  practice between programmes

4
A Collaborative Capacity Capability building
model

• Peer learning through Action Learning Set
• Support, advice and mentoring, F2F, e-mail and by
  phone (e.g. data analysis).
• See Burgess, Hilary and Carpenter, John (2008)
  'Building Capacity and Capability for Evaluating
  the Outcomes of Social Work Education (the OSWE
  Project) Creating a Culture Change', Social Work
  Education.

5
Measuring Outcomes

• Measuring changes over time
• Before-after designs
• Using and adapting measures

6
Primary levels of Outcomes (after Carpenter,
2005)

• Levels of Outcome
• 1.Learners reactions
• 2.Modifications in attitudes and perceptions
• Attitudes
• Motivational
• 3.Acquisition of knowledge and skills
• Procedural
• Strategic knowledge
• Initial skills
• Compilation skills
• 4. Changes in Behaviour
• 5. Benefits to users and carers

• Focus
• Attitudes to race and racism (Soton)
• Attitudes interprofessional practice (Sussex)
• Understanding partnership with SUC (Anglia
  Ruskin)
• Communication skills with children (Sussex)
• Interviewing and communication skills (York)
• Use of research skills (Bournemouth)
• Acquisition of Soc. Wk. competences (Oxford
  Brookes)

7
The outcomes of teaching and learning about
race and racism

• Kish Bhatti-Sinclair
• Division of Social Work Studies
• School of Social Sciences
• University of Southampton

8
Aims and Methods

• Students understanding and experience of race
  and racism and
• How race and racism is addressed in the degree
  curriculum
• Questionnaires to BSc social science students in
  week 2 of Semester 1 (153 students) and in Week 8
  (71 students) of Semester 2.
• 34 respondents completed both sets. Social work
  students 41.2.

9
Main sources of learningabout race and racism
(frequency)

• TYPES OF LEARNING
• Seminars 17
• Course units 16
• Personal study 9
• Shadowing soc wkr 4
• Peer Group 3
• Personal tutors 2

• SPECIFIC COURSE UNITS
  2 out of total of 8 (3 SW)
• Social problems and social policy - 6
• Sociology of everyday life - 2

10
Results

• A small increase in self-rated knowledge of
  racism 10.
• A increase in respondents confidence in
  challenging racism (but not statistically
  significant.)
• Overall results suggest that formal teaching
  about race and racism did influence knowledge
  although the impact was small.

11
Limitations

• Sampling not random.
• Results not generalisable must be interpreted
  with caution.
• Knowledge of racism based on self rating - not
  possible to obtain an objective measure of
  knowledge.

12
Outcomes of Using Research for Practice using
blended learning.

• Bournemouth UniversityAnne Quinney
• BA Social Work Year 2 students.

13
Methodology

• Pre-Post tests
• Measure Research Self Efficacy (RSE) scale
  (Holden et al., 1999). Subscales on research
  knowledge and skills. PLUS 5 items using
  computer and information technology.
• Analysis Paired t-tests for difference in mean
  scores.

14
Findings Research Knowledge and Skills

• How confident are you that you can successfully
  analyse basic quantitative and qualitative data?
• Scale 0-10

15
Findings using computer and information
technology

• How confident are you that you can successfully
  access research findings from research bodies,
  social work organisations, government departments
  etc (e.g. JRF, SCIE) using the internet?

16
Next Steps and Uses

• Do Research Self Efficacy scores and assignment
  marks correlate?
• Compare with another programme
• Uses
• Adjust the curriculum for (1) present students
  who have to do a dissertation in year 3 and (2)
  next years course.
• Student self-assessment what I need to learn.
• The RSE scale has been adopted by ESRC Researcher
  Development Initiative.

17
Outcomes of teaching about Partnership
Interprofessional Practice

• Imogen Taylor, Clare Ockwell, Suzy Braye
• University of Sussex
• Note Class-room based module does not include
  students from other professions.

18
Methodology

• Outcomes of learning and teaching on BA MA
  students attitudes
• Stage 1 pre module teaching
• Stage 2 post module teaching
• Stage 3 end of course
• Validated scales from University of West of
  England IPE programme
• Communication and teamwork (CT)
• Interprofessional learning (ILL)
• Interprofessional interaction (II)
• Interprofessional relationships (IR)

19
Mean ratings at start/end of module (BA)
20
Differences in attitudes by years of pre-course
experience
21
Next steps

• Complete analysis of BA/MA T1 and T2
• Collect and analyse end programme BA/MA data (T3)
• Compare with UWE IPE findings
• Introduce new comparator programme (any offers?)

22
Outcomes of Teaching and Learning communication
skills

• Juliet Koprowska
• University of York

23
Research question methodology

• Are communication skills improved through
  learning within the university?
• Methodology multiple-measure case study design.
• Sample year 1 BSc Social Work

24
Research methods

• Self-efficacy scale
• Recorded interview with service user-actor
• Self-evaluation of interview
• Service user-actor evaluation
• Assessment of video-recording
• Feedback from service users on placement
• T1 prior to teaching, T2 3 months later,
  after teaching, T3 9 months later, after first
  placement

25
(No Transcript)
26
(No Transcript)
27
Implications

• Students may lose confidence as they acquire
  knowledge and understanding
• Potential for formative assessment
• Effect of placement needs exploration
• Qualitative discussion with students and actors
  would deepen understanding
• Should tools become regular measures to increase
  participation in research, with retrospective
  permission?

28
Communication Skills with Children Young People

• Michelle LeFevre
• University of Sussex
• How does a Programme contribute to students
  development of confidence and competence in
  communicating with children and young people?

29
What and How are these learned?

• Skilled/effective communication is not just
  techniques and micro skills. Also underpinning
  knowledge, values and ethical commitments,
  personal qualities and emotional capabilities.
• Learned throughout the whole programme not just
  in focused skills teaching and also through
  other personal and professional experiences.

30
MethodologyProspective Evaluation 4 stages

• Questionnaire
• Students personal characteristics Have
  particular kinds of student learned most/least?
• Measuring self-confidence in communication with
  children at different stages.
• Which aspects of the programme facilitated your
  confidence and skills?
• Case Vignette tool
• Students demonstrate knowledge of planning,
  implementing, reviewing and reflecting on
  communication with children.
• More objective measure (though analysis is
  subjective?)

31
Key Findings

• Students confidence has increased.
• Direct practice in placement the most important,
  followed by Child Development
• All aspects of focused skills teaching were
  perceived as helpful to learning
• Knowledge of aspects of communication skills
  increased (often significantly) - but not in
  every domain.
• Still to analyse relation of characteristics to
  learning and mapping individual students through
  programme.

32
Learning about working in partnership with
service users

• Roxana Anghel Joanna Fox
• Anglia Ruskin University
• Exploring the use of Concept Mapping

33
Method Concept Mapping (CM)

• Schematic representation of an individuals
  understanding of a knowledge domain in the form
  of concepts meaningfully linked in propositions.
  
• Design
• CM and questionnaire at T1 (induction week BA)
  T2 (end year 1) T3 (end year 2)
• CM task, unstructured - Working in Partnership
  with Service Users
• Questionnaire most impacting learning
  opportunities.
• Analysis of CM
• Generate scores based on validated concept-links
• Raters included service user and researcher.

34
Concept Map at start of programme
35
Concept map at end of Year 1
36
Concept Map at end Year 2
37

• Findings Working in Partnership

Map Scores
38
Sources of Learning

• At T1 Ethics and Values observational practice
  Poverty, Social Exclusion and Social Work, and
  academic reading.
• At T2 Social Work with Children and Families,
  Social Work with Adults, practice placement
  Principles and Skills of Social Work academic
  reading and discussing with colleagues

39

• CM used on two other modules

• Module A Map scores correlated with marks

• Module A (Year 3 BA, 13 pairs) 46 T2 maps
  increased content and quality

Map Scores
Students

• Module B (Year 1 MA, 11 pairs) 72 T2 maps
  increased content and quality

• Module B Map scores and marks did not correlate

Map Scores
Students
40

• Discussion
• Small numbers, so the value of CM as summative
  method in assessing outcomes needs to be explored
  further
• Success depends on the clarity of instructions
  and purpose, familiarity and time available.
• Uses self-assessment, visual learning aid,
  formative assessment (identify gaps), as well as
  assessing outcomes.
• Being firmly based on meaningful learning CM
  can be applied to many area of social work.

41
Measuring the Acquisition of Competency
Sharon Vitali Oxford Brookes University
42
Methodology

• Cohort Sample BSc Social Work
• (n38 at beginning n34 at present 89)
• Longitudinal - Baseline to Graduation
• Repeated Measures Fixed Outcomes (National
  Occupational Standards Units) RICET
• Multiple Methods

43
Year One - Semester One
?

• Baseline
• December

RICET
?
?
RICET
PERSONAL LEARNING PLAN
44
Consider your current state of competence
(knowledge/skills) and indicate a score (from the
guide below) for each of the learning outcomes
listed.

• 0 Can not produce any evidence of competence.
• 1 Understands the learning outcome, but can
  produce only limited or no evidence of
  appropriate attempts to put it into practice.
  Much more knowledge/practice needed.
• 2 Understands, and can offer evidence of
  tentative attempts to integrate into current
  knowledge/skill base.
• 3 Demonstrates competence with some
  regularity.
• 4 Advanced understanding and demonstrating
  adequate level of integration of knowledge,
  skills, and appropriate application.
• 5 Clearly understands and demonstrates
  consistent and appropriate application of
  knowledge and skills in practice.

45
Academic Average 61.4
Video Scores SU1.8 ER2.2
46
Year Three - Semester Two

• May (70 DAY PLO)

RICET
PRACTICE PORTFOLIO TUTOR EVALUATION
CRITICAL CAREER REVIEW TUTOR EVALUATION
PRACTICE PORTFOLIO P/T EVALUATION
RICET
47
PA End Place2 ST End Place2 PA Mid Place2 ST Mid
Place 2 PA End Place1 ST End Place1 PA Mid
Place1 ST Mid Place1 ST End Yr1 S2 Tutor End
Yr1S1 ST End Yr1 S1 ST Baseline
48
Types of Data
NOW end
49
Overview of the findings from projects increases
in

• Attitudes to race and racism - but not to
  self-confidence in challenging racism.
• Self-efficacy following learning (e.g. research
  skills) Do these correlate with marks?
• Conceptual understanding (C Maps, communication
  vignettes).
• Ratings of Competence (RICET)

50
But

• No measured increase in Communication skills
• No overall increases in attitudes to partnership
  working
• Measures do not necessarily correlate with marks.
• And we need to strengthen research designs,
  especially comparative studies.

51
Some Questions for Discussion

• What methods of learning and teaching should we
  compare? Are you interested?
• What is their potential for use in formative
  evaluation and self-evaluation by students?
• Should we use these measures routinely (like
  Oxford Brooks) to assess student outcomes?
• Are you interested in replicating and developing
  these methods?

52
Key issues

• Formulating an appropriate, specific research
  question
• Selecting/adapting appropriate methods/measures
  for measuring change
• Negotiating student engagement, balancing ethical
  concern to allow opt-out with maximising
  engagement to promote learning
• Establishing involvement of service users/carers
• Getting support from colleagues
• Freeing space/time/resources for the research

53
Key issues (cont)

• Balancing roles of teacher and researcher
  (sometimes complementary, sometimes conflicting)
• Accessing expertise/learning to analyse and
  interpret data
• Finding comparison sites
• Feedback to programme/colleagues about
  knowledge/skills gained and implications for rest
  of programme

54
Thank you

• To our sponsors SCIE, SWAP and IRISS
• Contacts j.s.w.carpenter_at_bristol.ac.uk
• h.c.burgess_at_bristol.ac.uk
• Resource
• Evaluating Outcomes in Social Work Education
• http//www.iriss.ac.uk/node/88