Multidimensional Assessment for eLearning Traditional and Social Assessment Models for Student Persi

1
Multi-dimensional Assessment for eLearning-
Traditional and Social Assessment Models for
Student Persistence
Thomas E. Downey, Ph.D., MBA Chief Academic
Officer Embanet ULC
2
Urgency About Student Persistence

• Embanet partners with Universities to bring their
  programs online
• Over 53 programs launched
• Over 750 courses developed
• Over 20 Knowledge Partners
• Economics of eLearning - Student Persistence is
  a shared goal between universities and business
• Quality of the learning environment
• High graduation rate
• Increased revenue

3
A Few of Our Education Partners
4

• Student Retention
• students are more likely to persist when they
  find themselves in settings that hold high
  expectations for their learning, provide needed
  academic and social support, and actively involve
  them with other students and faculty in learning.
  The key concept is that of educational community
  and the capacity of institutions to establish
  educational communities that involve all students
  as equal members.
• - Tinto, 2002

5
Tintos Persistence Research
6
Student Persistence

• eLearning historically has a high drop-out rate
  / low persistence
• You cant manage what you cant measure
• Measurements are needed to allow us to track and
  intervene in a timely manner.
• Longitudinal measures are needed for long term
  Continuous Quality Improvement (CQI) and
  accreditation data.
• Timely data where interventions and improvements
  can yield positive results for students
• Multiple measures at multiple levels yielding
  actionable recommendations

7
Student Persistence Measurement

• The eLearning environment presents unique
  challenges in assessment
• Pedagogical methods have changed
• Objectivist vs. Constructivist Teacher vs.
  Coach
• Asynchronous vs. synchronous Time for
  reflection
• Individual vs. virtual group Communication and
  PM
• Virtual presence person-to-person and
  person-to-group interaction Communities of
  Inquiry (COI)
• Traditional models are not sufficient

8
Relevant eLearning Assessment

• Assessment of elearners should more closely match
  the environment that they will use for their
  professional careers
• eLearners must interact in disassociated
  environments
• Community is virtual, temporary, focus on task is
  tight, time schedules are short, teamwork and
  communication skills are critical
• Everyone must contribute and collaborate
• How do we assess this today?

9
Defining Student Persistence

• Community of Inquiry (CoI) (Swan, Richardson
  et.al., 2007)
• Social Presence
• Cognitive Presence
• Teaching Presence
• Entrepreneurial Learner of the 21st Century
  (Reimers, King, et.al, 2005)
• Internal Locus of Control (LOC)
• High need for achievement
• Moderate risk-taking propensity
• Interaction in Courses (Chickering and Ehrmann
  1996)
• Instructor involvement / feedback / student
  interaction
• Drives longer persistence in course / program
• Emotional Intelligence (Berenson, Boyles,
  et.al.,2008)
• Learners emotional response to the learning
  environment
• Self-awareness of ones own feelings / needs

10
Missing Dimensions

• We must measure effectiveness of the total
  learning environment not limited to the course
• We need an inclusive and comprehensive assessment
  model
• Above the course level to the enterprise level
• A systematic / proven approach
• Look to business / IT
• Service Level Agreements (SLAs)
• Quality of Service (QOS)
• Continuous Quality Improvement (CQI)

11
Missing Assessment Dimensions

• We require measurements for a dynamic eLearning
  system mirroring the virtual work environment
• Group work how do we create a fair model of
  assessment
• 360 degree assessments
• Longitudinal assessment - measuring goal
  completion
• ePortfolios with feedback from the instructor
• Reflections on learning activities

12
Sources of Multi-Dimensional Data

• Students
• Program Administrators
• Student Services Advisors
• Technical Support Services
• Service logs / types of issues / resolution
• Instructional Designers
• Instructors
• Facilitators
• Student grades

13
Multi-Dimensional Data Gathering

• Multi-Part and Multi-Role Surveys
• Web-based Survey Software
• Common core of questions / differentiated
  questions by role
• Survey Participants
• Students
• Instructors / Facilitators
• Instructional Designers
• Program Administrators
• Student Services Advisors
• Student Grades
• Technical Support Logs

14
Levels for Reporting
Longitudinal Reports / Measurements Over Time
Enterprise Level
Program Level
Course Level
Student Level
15
Sources of Data by Level
Longitudinal Reports / Measurements Over Time
eScore Card - Course / Program / ePortfolios /
Changes Implemented effect on scores
Enterprise Level
System Support up time, types of issues
Program Level
Program Administrators, Students
Course Level
Students, Facilitators, Instructors, Student
Services, Student Grades
Student Level
End of Course Critique, Grades, Time on Task,
Involvement / Interaction
16
Benefits of These Reports
Longitudinal Reports / Measurements Over Time
CQI and QOS - Over Time, Accreditation Data,
Repositories of Exemplar Work
Enterprise Level
Systems Performance, Technical Support, Student
Entry Level need for training
Program Level
Administrator Satisfaction (Program Goal
Attainment), Student Persistence
Course Level
Ss Performance, Design of Course, Persistence,
Effectiveness of Course / Ss Support
Student Level
Student Involvement / Interaction / Affect /
Satisfaction
17
M-D Assessment Project Status

• Project Partners Identified
• Multi-part survey in testing
• Tech Support log summary reports completed
• Next Steps
• Field test data gathering with live data
• Create test reports
• Review with Knowledge Partners for Completeness
  and Accuracy
• Roll out to larger segment of Knowledge Partners
• Incorporate into Suite of Value Added Features
  from Embanet

18
Thank You.