Collaborative Evaluation of an Online Graduate Nursing Curriculum at the University of Minnesota

1
Collaborative Evaluation of an Online Graduate
Nursing Curriculum at the University of Minnesota

• ISSOTL 2005

Melissa Avery Bradley A. CohenJ.D. Walker
2
Presentation Outline

• Project Background
• Evaluation Goals
• Evaluation Process
• Evaluation Instrument
• Key Results
• Next Steps

3
Project Background

• Part of the evaluation of TELIGN project, a large
  federally funded project to create hybrid
  curriculum in SoN
• Formative evaluation of 16 hybrid graduate level
  courses
• Collaboration between SoN faculty and DMC
  instructional designers

4
Project Background

• Animating considerations
• Existing general evaluation instruments are
  unlikely to capture SoN values
• Evaluation instruments for online courses should
  avoid instructional design jargon
• Transferring F2F peer review practices to an
  online environment may be difficult for faculty
  members

5
Evaluation Objectives

• Identify or generate tools/processes that reflect
  values of SoN and that can be used to evaluate
  hybrid nursing courses
• Identify best practices with respect to course
  design and implementation of graduate level
  hybrid nursing courses
• Create a peer review process for evaluating
  online teaching that mirrors F2F peer review
  process in SoN

6
Evaluation Objectives

• Evaluate 16 courses with attention to
  instructional design, critical technical issues
  and implementation
• Develop a process, including tools, that faculty
  can use to design good online courses
• Provide a foundation for maintaining and
  improving quality online instruction in SoN
• Disseminate results

7
Activity

• With a partner, try to come up with at least two
  or three characteristics of an online learning
  environment you believe ought to be evident in a
  quality course.

8
Characteristics of excellence
9
Evaluation categories generated by our process

• Mechanics of course
• Student support
• Course organization
• Communication

10
Evaluation Process

• Generate evaluation objectives (lit review and
  several meetings)
• Identify the elements that the School of Nursing
  regards as essential in quality online courses

11
Evaluation Process

• Develop evaluation instrument within three
  constraints
• Reflects values generated in (2) above
• Does not require an excessive amount of time to
  complete when applied to a course
• Aids faculty in the improvement of existing
  courses and the development of new courses
• Refine instrument with inter-rater reliability
  test on representative course and willing
  instructor

12
Evaluation Process

• Evaluate 16 online courses using instrument (one
  instructional designer and two School of Nursing
  faculty members evaluate each course)
• Debrief the instructors of those courses (one
  instructional designer or one School of Nursing
  faculty member interviewed each instructor)

13
Evaluation Process

• Analyze the data collected
• Disseminate results

14
Questions about Objectives or Process?
15
Instrument and Data

• Evaluation instrument
• Results
• Quantitative
• Qualitative
• Actions taken

16
Evaluation Instrument

• Inspired by work done under the Partnerships for
  Training program, a national initiative funded by
  The Robert Wood Johnson Foundation.
• Requires roughly 2-4 hours per course

17
Evaluation Instrument

• 20 scale items, 1 open-ended, space for comments
• 5-point rating scale

18
Evaluation categories

• Mechanics of course
• Student support
• Course organization
• Communication

19
Mechanics of course

• Objectives clear
• Objectives appropriate
• Pre-requisites
• Time commitment
• Objectives match activities
• Technical requirements

20
Student support

• Faculty available
• Technical support
• Activities described clearly
• Support for learning activities
• Course layout

21
Course organization

• Student input
• Reflection
• Simple to complex
• Activities for multiple learning styles
• Evaluation clear

22
Communication

• Faculty timely responses
• Student-faculty interaction
• Student-student interaction
• Course interactivity

23
Key Results Quantitative

• Lowest mean ratings
• Highest mean ratings
• Items with greatest variability across courses
• Inter-rater variability

24
Key Results QuantitativeHighest and lowest
ratings
Overall mean 3.91
25
Key Results QuantitativeLowest ratings

• Q4b There is a written connection between the
  course objectives and learning activities (M
  2.88)
• Q7 Technical requirements are specified with
  regard to both skill and equipment (M 3.16)
• Q8 Realistic time commitment related to credit
  load (M 3.26)
• Q3 Pre-requisite or prior knowledge required for
  course outlined (M 3.38)
• Overall mean 3.91

26
Key Results QuantitativeLowest ratings

• possible explanation most courses used
  boilerplate for these items (e.g., "graduate
  student in nursing" for Q3)
• conclusion a decision for the School of Nursing?

27
Key Results QuantitativeHighest ratings

• Q2 Goals and objectives appropriate to level of
  the course (M 4.51)
• Q1 Goals and objectives clearly stated (M 4.40)
• Q9 Learning activities are clearly described (M
  4.40)
• Q10 Learning activities applicable to the course
  are sufficiently supported (M 4.34)
• Q12 Evaluation mechanisms/instruments measure
  objectives (M 4.28)
• Q4a Learning activities and teaching strategies
  match the course objectives (M 4.26)
• Overall mean 3.91

28
Key Results QuantitativeRating variability
29
Key Results QuantitativeRating variability

• Q8 Realistic time commitment related to credit
  load (SD 1.56)
• Q3 Pre-requisite or prior knowledge required for
  course outlined (SD 1.34)
• Q4B There is a written connection between the
  course objectives and learning activities (SD
  1.29)
• Q19 Faculty and student interaction is timely
  and appropriate (SD 1.23)
• Q7 Technical requirements are specified with
  regard to both skill and equipment (SD 1.21)

30
Key Results QuantitativeRating variability

• possible explanation for several items,
  reviewers may have applied the item criteria
  differently
• conclusion disambiguate certain items on
  evaluation instrument work on inter-rater
  reliability

31
Key Results QuantitativeInter-rater variability

• DMC reviewers gave higher mean ratings than
  Nursing reviewers on 16 out of 21 items
• these differences were statistically significant
  in 4 cases (Q2, Q5, Q12, Q15)

32
Key Results QuantitativeInter-rater variability

• Possible explanations
• Nursing faculty more knowledgeable, hence more
  critical
• DMC staff are invited outsiders, hence less
  critical
• DMC staff have seen many online courses of widely
  varying quality

33
Questions about quantitative data?
34
Key Results QualitativeMethodology

• Three PIs independently reviewed compiled
  comments from all reviewers of all courses, and
  debriefing transcripts
• Special attention paid to
• best practices
• recurring themes
• disagreements among reviewers
• apparent confusion regarding questions

35
Key Results Qualitative

• Three action categories emerged
• Course-specific items for improvement by
  individual faculty
• Issues for School of Nursing faculty to address
  as a group
• Items that reflect a need to improve evaluation
  instrument

36
Key Results Qualitative

• Q4 Learning activities match course objectives
• Findings
• sometimes unclear match between activities and
  higher-level objectives
• explicit connection rare

37
Key Results Qualitative

• Q4 Learning activities match course objectives
• Actions
• faculty use more active strategies, show
  connection to higher-level objectives
• School of Nursing is explicit connection
  necessary?

38
Key Results Qualitative

• Q13 Course content appeals to a variety of
  learning styles
• Findings
• excellent use of varied learning activities
• inconsistent use of varied media

39
Key Results Qualitative

• Q13 Course content appeals to a variety of
  learning styles
• Actions
• evaluators promulgate best practices
• evaluators rewrite question to distinguish
  between varied media and varied activities

40
Key Results Qualitative

• Q18 Student to student interaction built into
  learning activities
• Findings
• use of online communication tools common
• genuine student-student interaction inconsistent
• instructions on how to collaborate, work in
  groups, etc. inconsistent

41
Key Results Qualitative

• Q18 Student to student interaction built into
  learning activities
• Actions
• faculty look to best practices for guidance on
  fostering interaction
• evaluators rewrite question to distinguish among
  different aspects of interactivity within a
  course

42
Questions about Qualitative Data?
43
Actions taken

• Revised evaluation tool
• Created rubric to increase inter-rater
  reliability
• Made tools, results available on DMC web site
• Presented results to School of Nursing
• Consulted with Department of Family Education
  regarding similar process
• Promoted use of evaluation tool as guide for
  faculty preparing to teach online
• Identified and publicized best practices

44
Technology Enhanced Learning In Graduate
Nursing
45
Best Practice Examples

• Case studies
• Self tests
• Quizzes
• Discussion
• Peer review activity
• Video

46
Case study example

• Sarah, 16-year-old, primigravida at 40 weeks. Her
  mother takes the phone and states "something has
  to be done." Sarah has been up all night with
  irregular contractions, vary in intensity, every
  6-15 minutes. Yesterday her cervix was long,
  thick, closed with the vertex at 0 station.
• What further information do you wish to have?
• Develop two different plans of care for her at
  this particular point and give your rationale for
  each.
• Do you feel that the plan would vary depending
  upon your practice situation (hospital based
  in-house call versus home call).

47
Pharmacology case study

• Sandra presents with mild dehydration, a
  productive cough with purulent sputum for three
  days, low-grade fever of 99 - 101. She did not
  sleep well last night because of cough, SOB
  taking the stairs and going out to get the mail.
  Very fatigued, has not cared for children past
  two days. Asthma since childhood, chronic
  bronchitis (last treated 1 month ago with
  Amoxicillin), and type 2 diabetes (oral
  hypoglycemics past 5 years). She smokes 1
  pack-per-day.

48
Clinical decision making

• PROBLEM IDENTIFICATION
• What information indicates the presence of
  pneumonia in this patient?
• THERAPUTIC ALTERNATIVES
• What alternatives are available for the treatment
  of this patient?
• OPTIMAL PLAN
• What kind of treatment plan would you advise for
  this patient?

49
Clinical decision making

• ASSESSMENT PARAMETERS
• What clinical and lab parameters can be used to
  assess the resolution of the pneumonia and
  success of the medication therapy?
• PATIENT COUNSELING
• What important information concerning therapy
  should be provided to this patient?

50
Prescription template

• Name DOB (use stated age) Date
• Rx
• Disp
• Sig
• Refill Signature
• DEA
• Substitution Permitted
• Not Permitted
• Rationale for choice

51
N8503 Self test correct answer
52
N8503 Self test incorrect answer
53
N5200 quiz question
54
N5222 self quiz with pop-up feature
55
Peer activity in research

• What is the relationship between geographic
  demographics and childhood obesity? DESIGN
  descriptive correlational SAMPLING Instead of
  children age 6-12, I am thinking just 1st and 2nd
  grade. Simple random sampling. VARIABLES
  height, weight, BMI interview child with parent,
  questions related to activity level, nutrition
  education, parental job/education, meal choices,
  health education.

56
Peer input

• What is the reason for the age group 7-8 year
  olds? Any consideration for factors such as the
  nutritious value of school provided breakfast and
  lunch for those children on these programs vs.
  children who eat at home? Other variables to
  consider are physical ed programs at school, the
  childs activity level e.g. child walks to and
  from school daily, for some children across the
  street, for others more than a mile each way.

57
Use of video and audio/ppt

• Short clips
• Several portions
• Demonstrations
• Avoid the talking head
• Audio/Powerpoint

58
N8503 Breastfeeding video
59
Talking Circle

• The Talking Circle (also Talking Story or Sharing
  Circle) is an ancient tool used by American
  Indians to carry out a group process. Many
  indigenous populations used the Talking Circle as
  a culturally appropriate way to collectively
  convey information among Circle participants with
  thoughts of utmost respect in mind. The Talking
  Circle customarily uses oral tradition as the
  foundation to relay information to its
  participants. Everyone has an opportunity to
  contribute to conversation and group process.

60
Discussion instructions

• Online "attendance" required. Log on, post a
  substantive contribution at least twice a week.
• 30 substantive postings to questions assigned to
  your group during the semester for full credit.
• One posting due by midpoint, one by end.
• Connect the concepts in the readings with your
  practice, your specialty, and your philosophy
• Response brief, thoughtful (100-250 words).
• Welcome responses that agree and disagree.

61
Additional strategies

• Place external links in a convenient spot for
  students to use, within content modules
• Detailed descriptions about time commitment for
  course participation
• Encourage student reflection on own learning
• Detailed descriptions of assignments
• Scaffolding building from simpler to more
  complex learning activities throughout course

62
Next steps - SoN

• Develop a peer review tool
• New grant funded
• Adding interactivity to courses
• Continued sharing of activities

63
Samples of newly developed interactive learning
activities

• Welcome message
• Type, click and tell
• Flash animation
• Games
• Course closure

64
Acknowledgement

• This project was supported by funds from the
  Division of Nursing (DN),Bureau of Health
  Professions (BHPr), Health Resources and Services
  Administration (HRSA), Department of Health and
  Human Services (DHHS) under grant number 6
  D09HP00115-03-01, NM, WHCNP, and PHN Graduate
  Education via Technology and D09HP04068-02-01
  Technology Enhanced Learning in Graduate
  Education.  The information or content and
  conclusions are those of the authors and should
  not be construed as the official position or
  policy of, nor should be any endorsements be
  inferred by the Division of Nursing, BHPr, DHHS
  or the U.S. Government.

65
Collaborative Evaluation of an Online Graduate
Nursing Curriculum

• THANK YOU!
• Melissa D. Avery avery003_at_umn.edu
• Bradley A. Cohen cohenb_at_umn.edu
• J.D. Walker jdwalker_at_umn.edu
• Materials online at
• http//dmc.umn.edu/nursing-evaluation/