A Comprehensive Approach to Designing Online Courses - PowerPoint PPT Presentation

About This Presentation
Title:

A Comprehensive Approach to Designing Online Courses

Description:

Title: A Comprehensive Approach to Designing Online Courses Author: jfreeman Last modified by: iBAHN Created Date: 12/6/2007 7:50:44 PM Document presentation format – PowerPoint PPT presentation

Number of Views:245
Avg rating:3.0/5.0
Slides: 34
Provided by: jfreeman
Category:

less

Transcript and Presenter's Notes

Title: A Comprehensive Approach to Designing Online Courses


1
ACADEMIC IMPRESSIONS
Online Course Design
Jennifer Freeman
EVALUATION
This session will focus on the formative,
summative, and confirmative evaluation of online
instructional materials. Methods and tools for
defining, quantifying, and measuring quality in
online courses will be presented. Participants
will use provided templates, tools, and rubrics
to develop a sample evaluation plan and critique
an online course.
2
Evaluation Session Goals
  • Understand the difference between assessment and
    evaluation
  • Define formative, summative and confirmative
    evaluation and understand the importance of each
  • Explore theories and methods of evaluation
  • Design a basic revision plan

3
Evaluation vs. Assessment
  • Evaluation measuring the quality and
    effectiveness of learning materials and
    activities
  • Assessment measuring students learning and
    achievement of goals and objectives

4
Course Evaluation Why?
  • Fix things that are broken
  • Ensure learning outcomes are being achieved
  • Discover causes for failuresproactively fix
    other problems
  • Discover potential usability/accessibility issues
  • What works in theory doesnt always work in
    practice
  • Constant maintenance and improvements to content
    and strategies
  • Dynamic nature of online learning

5
What Do We Evaluate?
  • Instructional materials alignment with
    objectives and the effectiveness of testing
    instruments
  • Quality of instructional materials
  • Quality of external resources
  • Effectiveness of instructional strategies
  • Usability of tools and technology
  • Effectiveness of teaching skills
  • Evaluation chart

6
Formative Evaluation of Instructional Materials
  • Why?
  • Uncover problems early on fix broken stuff
    (hopefully before students find it)
  • Identify potential usability/accessibility issues
  • Examine effectiveness and improve functionality
  • Dynamic nature of online learning
  • What? When?
  • An ongoing process, usually done both during
    development and while being taught
  • Asks the question, How are we doing?

7
Formative Evaluation of Instructional Materials
  • Who will use this evaluation information?
  • Course development team
  • Instructor
  • How? What should be evaluated?
  • Instructional materials
  • Instructional strategies
  • Use of tools and technology

8
Formative Evaluation Questions to Ask
  • Do learning activities and assessments align with
    the learning objectives?
  • Do learning materials meet quality standards?
  • Are learning materials error-free?
  • Are learning materials accessible?
  • Are learning materials usable?
  • Are the technology tools appropriate and working
    properly?

9
Formative Evaluation Gathering data
  • Course development rubrics
  • Checklists
  • Focus group feedback
  • For consistency and repeatability
  • To reduce bias, evaluation should not be done by
    members of the course development teamyou need a
    fresh pair of eyes
  • Develop a testing process/protocol/list of
    questions/evaluation forms
  • Determine a standardized, set time for testing so
    it occurs regularly as a normal part of the
    process
  • Help desk error logs
  • Student FAQ discussion threads
  • Extra credit for errors found idea
  • Faculty notes jotted down during semester

10
course is released
11
Sample Formative Evaluation Methods
  • Checkpoint 1 syllabus, outline and first
    lesson
  • Checkpoint 2 half of the course, viewed on
    multiple platforms
  • Checkpoint 3 Entire course proofread/edited
  • Checkpoint 4 Entire course Q/A checked
  • Checkpoint 5 Final check (previous errors)
  • Student survey after first three lessons
  • Instructor survey after first three lessons
  • Examination of help desk error logs

During development
Course is Live
12
Summative Evaluation
  • Why?
  • Examine effectiveness
  • Improve functionality
  • Discover causes for failuresfix existing
    problems
  • What works in theory doesnt always work in
    practice
  • Constant maintenance and improvements to content
    and strategiesthe dynamic nature of online
    learning
  • What? When?
  • Usually done after the completion of each
    semester
  • Asks the question, How did we do?

13
Summative Evaluation
  • Who will use this evaluation information?
  • Instructor
  • Course development team
  • Administration
  • How? What should be evaluated?
  • effectiveness of instructional materials and
    strategies
  • the learning environment
  • the instructors teaching skills
  • availability and ease of use of tools and
    technology
  • instructor satisfaction with the online teaching
    experience
  • student satisfaction with the online learning
    experience

14
Summative Evaluation Questions to Ask
  • Did the students succeed? (grades)
  • Did the learning activities and assessments align
    with the learning objectives?
  • Were assignments and assessment appropriate to
    the content?
  • Was time adequate to convey material and complete
    tasks?
  • Level of instructor and student satisfaction
    (participation and opinion)
  • Were learning materials easy to use and
    accessible? What content did students frequently
    have problems with? What areas of the course are
    error-prone?
  • Were there any concerns about motivation?
  • What tools did the instructor or students
    frequently have problems with? Should we continue
    to use chosen tools?
  • Are program/department needs being met
  • accreditations, prerequisites for other courses,
    competencies
  • Is the course scaleable?

15
Summative Evaluation Gathering data
  • Student grades
  • Student surveys
  • Instructor satisfaction surveys
  • Learner self-assessments
  • Pretest/posttest comparisons
  • Assessment item analysis
  • Focus group feedback
  • Help desk error logs
  • Discussion forum and chat archives

16
Confirmative Evaluation
  • Why?
  • Discover long-term effectiveness of the course
  • Address large-scale changes necessary to the
    curriculum
  • Constant maintenance and improvements to
    technology, content and strategiesthe dynamic
    nature of online learning
  • What? When?
  • Usually done some time after the completion of
    each semester
  • Asks the question, How are we doing now?

17
Confirmative Evaluation
  • Who will use the evaluation information?
  • Instructor
  • Administration
  • How? What is being evaluated?
  • Students long-term retention of learning,
    usefulness to their long-term goals
  • Long term effectiveness of the course within the
    program
  • LMS and other technology/tools

18
Confirmative Evaluation Questions to Ask
  • Are program/department needs being met
    (accreditations, prerequisites for other courses,
    competencies)
  • Trends in level of student satisfaction
  • Course valuable/meaningful to long-term goals?
    (program/career)
  • Is the course scaleable?
  • Is the course sustainable?
  • Learning environment, technology, tools still
    meeting our needs?

19
Confirmative Evaluation Gathering data
  • Program student surveys
  • Departmental administrative opinions
  • Faculty peer review of learning materials
  • Employer surveys
  • Retention data
  • Help desk logs
  • LMS effectiveness study / survey

20
Revision Resources Sample Formative Evaluation
and Revision Plan
  • Analysis of problems found
  • How urgent is it?
  • How long will it take to fix?
  • Assign each issue a priority score
  • Establish a threshold below which a course will
    be postponed
  • Prioritized list of change requestswhens the
    best time to revise?
  • Assign corrections and establish a deadline for
    each
  • Make note of unaddressed issues

21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
Sample Summative Evaluation and Revision Plan
  1. Analyze student and faculty surveys identify
    themes or trends
  2. Analyze assessment
  3. Analyze help desk logs
  4. Examine course archives
  5. Compile list of issues (including issues noted
    during formative phase that have yet to be
    addressed)
  6. Research solutions
  7. Determine time needed to fix
  8. Assign priority ratings
  9. Assign tasks and establish deadlines

25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
Revision ActivityMusic Appreciation
  • Ten modules, each includes lecture notes, one or
    more Powerpoint presentations with audio, a
    graded listening quiz and a synchronous chat
    requirement
  • Project My Favorite Composer biography
    assignment submitted as Word doc to instructor
  • Midterm multiple choice exam
  • Final multiple choice exam

30
Common LMS Evaluation Criteria
  • Costs rising at a reasonable rate?
  • Are server space and maintenance needs being met?
    Is the vendor and software in compliance with
    required standards?
  • How reliable has the system been?
  • Have there been any security concerns?
  • Level of customization possible within the
    system?
  • Satisfied with the structure and presentation of
    courses?
  • Satisfied with the authoring tools provided?
  • Satisfied with the tracking capabilities of the
    system?
  • Satisfied with the testing engine and/or
    assessment tools available in the system?

31
Common LMS Evaluation Criteria
  • Satisfied with the collaboration tools
    (discussion areas, journaling, help desk,
    whiteboard) provided through the system?
  • Satisfied with the productivity tools (calendar,
    help files, search engine) provided through the
    system?
  • Is student / faculty / staff documentation or
    training sufficient?
  • How usable do students, faculty and staff find
    the tools?
  • What is the vendors reputation in the industry?
  • What is the vendors position in the industry?

32
What Weve Learned
  • The difference between assessment and evaluation
  • Definitions of formative, summative and
    confirmative evaluation and the importance of
    each
  • Methods of evaluation
  • Sample evaluation and revision plans

33
  • Jennifer Freemanjfreeman_at_utsystem.edu
Write a Comment
User Comments (0)
About PowerShow.com