Transiting to an Online Course Evaluation Model: The Online Student Survey of Instruction - PowerPoint PPT Presentation

Loading...

PPT – Transiting to an Online Course Evaluation Model: The Online Student Survey of Instruction PowerPoint presentation | free to download - id: 502d5c-Njk3N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Transiting to an Online Course Evaluation Model: The Online Student Survey of Instruction

Description:

Florida Association of Institutional Research 2010 Annual Conference Transiting to an Online Course Evaluation Model: The Online Student Survey of Instruction – PowerPoint PPT presentation

Number of Views:189
Avg rating:3.0/5.0
Slides: 46
Provided by: Coraggi1
Learn more at: http://www.spcollege.edu
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Transiting to an Online Course Evaluation Model: The Online Student Survey of Instruction


1
  • Florida Association of Institutional Research
  • 2010 Annual Conference
  • Transiting to an Online Course Evaluation Model
    The Online Student Survey of Instruction

February 2010
2
Student Survey of Instruction
  • Presenters
  • Dr. James Coraggio, Director, Academic
    Effectiveness and Assessment
  • Magaly Tymms, Assessment Coordinator, Academic
    Effectiveness and Assessment
  • With contributions from
  • Debbie Montalvo, SSI Coordinator
  • Richard Jakubowski, SSI DB Administrator

3
SPC Background
  • SPC, established in 1927, is the oldest
  • 2-year college in Florida
  • First Community College in Florida to offer 4
    year degrees (2002)
  • Nine Campuses throughout Pinellas County
  • 2008-09 FTE 16,967 (LD), 1,677 (UD)
  • Opening Fall 2009 credit enrollment 29,560
  • Annual 2008-09 headcount (credit and non-credit)
    61,414

4
Purpose
  • This presentation describes the development and
    implementation of a college-wide online process
    to administer course evaluations, the Student
    Survey of Instruction (SSI), which takes
    advantage of existing technology to efficiently
    collect student responses, disseminate the
    results in a more timely manner, and save the
    institution time and money.

5
Agenda
  • Issues with the prior process
  • Intended outcomes
  • New online process
  • Dissemination of results and reports
  • Response rates
  • Next steps

6
Issues with Prior Process
  • Expenses related to Scantron forms, and numerous
    hours of sorting, organizing, and transporting
    forms
  • Time consuming process of transcribing open-ended
    items
  • Not all Scantrons were scanned and processed
    (scanning issues, data integrity issues, etc.)
  • Scantron results and online results captured
    separately and reported separately
  • Process conducted by multiple departments with no
    central ownership

7
Original SSI Items
  • Original SSI forms were developed by a college
    committee a number of years ago
  • Items were reviewed for content and a lecture
    form was aggregated into four categories (e.g.,
    faculty student interaction, organization,
    presentation, and evaluation)
  • There were four SSI forms (Scantron) Lecture,
    Non-lecture, Clinical A, and Clinical B
  • Results were scanned by a third party (PCSB) and
    entered into PeopleSoft

8
eCampus SSI Items
  • Online SSI form was developed later by eCampus to
    assess online courses
  • This form was similar to the Lecture SSI form
    with some slightly modified items and the same
    four categories
  • Original online survey tool created to capture
    summative information in database

9
Performance Improvement
  • From Compliance to Performance Improvement

10
Intended Outcomes
  • New SSI online process was designed to
  • Provide stakeholders (Faculty, Program
    Administrators, etc.) information to assist in
    making decisions that will result in performance
    improvement
  • Provide more reliable and valid course evaluation
    information
  • Create centralized coordination of the process
  • Save the college time and money (est. 45,000 per
    year)
  • Eliminate the need for in-class administration
    (more course time for instruction)

11
SSI Online Process Timeline
  • Fall 2007
  • Established an SSI Team
  • Factor analysis (exploratory and confirmatory
    cross validation) determined that lecture and
    online SSI forms had three factors (e.g., faculty
    engagement, preparation and organization, and
    course instruction)
  • Spring 2008
  • Pilot of 100 FTF course sections conducted
  • Fall 2008
  • First collegewide administration (All course
    ANGEL shells enabled)
  • Created an online reporting tool (program
    administrators only)
  • Revised older survey administration tool

12
SSI Online Process Timeline
  • Spring 2009
  • Created customized usage reporting tools
  • Created new survey interface with an opt-out
    capability
  • Added Faculty Representative to SSI Team
  • Created online reporting tool access for Faculty
    (No more paper)

13
SSI Online Process Timeline
  • Summer 2009
  • First collegewide summer administration
  • Modified faculty email notices to include link to
    real-time response data
  • Began working with HEC on Clinical Solutions
  • Evaluated current SSI items and forms (SSI
    Revision Committee)
  • Fall 2009
  • Implement self-service module so faculty can
    revise their own administration dates

14
SSI Online Process Timeline
  • Spring 2010
  • College-wide Pilot of new SSI items set
  • Pilot Clinical SSI in the areas of MLT, EMS and
    COE
  • Add standard deviation information to reports
  • Identify best practices from faculty members and
    programs with the highest response rates

15
SSI Link in ANGEL
  • Course instructors are required at the start of
    each term to
  • enable their ANGEL course shells,
  • upload their course syllabus, and
  • attached their SSI link into their course.

16
Faculty Notification Emails
  • Faculty are provided four informational emails
    during their SSI administration.
  • Five days before the beginning
  • At the 50 mark
  • At the 75 mark
  • Two days after the end
  • Emails contain course, form, date, and response
    information.

17
Faculty Notification Emails
18
SSI Link in ANGEL
19
SSI Link in ANGEL
20
SSI Link in ANGEL
21
Accessing Information
  • A site was developed to provide faculty and
    program administrators information about the SSI
    online process http//www.spcollege.edu/central/s
    si/

22
Information about the Process
23
Information about the Process
24
Accessing Results
  • All SSI survey reports are available on the
    online SSI Administration site
    http//it.spcollege.edu/ssi/
  • Faculty and Program Administrators can login by
    entering their SPC network username and password
  • Access is limited to only those courses under
    their area of responsibility.

25
Accessing Results
26
Accessing Results
  • Instructors and Program Administrators can select
    Term from the drop-down menu (i.e. 0405) and
    select how they would like to view the results
  • Each individual section with its own results OR
    Combined sections by instructor

27
Accessing Results
  • After selecting a term and indicating Each
    Individual Section with its own results, users
    can view each individual section with its own
    results, as selected on the previous screen for
    Term 0405.
  • By selecting any of the courses, users can view
    the SSI report for that course.

28
Accessing Results
29
Accessing Results
  • For the Combined sections by Instructor
    options, users can view combined sections with
    results, as selected on the term selection screen
    (Term 0405 in this example).
  • By selecting any of the instructor names, users
    can view a combined SSI report for that
    instructor.
  • Note multiple reports for a single instructor
    may be listed if the instructor taught courses
    with either different SSI survey forms, different
    academic orgs, or on different campuses.

30
Accessing Results
31
Accessing Results
  • For an electronic version of the report (pdf),
    users can select View in Adobe Reader
  • Each Individual Section with its own results
    report produces individual SSI results, three
    comparative measures, graph, and student
    comments.
  • Combined sections by Instructor report produces
    an instructors combined sections SSI results
    with three comparative measures, graph, and
    student comments by course. This report is
    somewhat similar to the SSI reports generated by
    AIS in the past.

32
Administration Dates
  • SSI is intended to be an evaluation of the
    instruction provided during the course.
  • It is administered after the students have
    participated in the majority of the course (no
    sooner than the 60 mark of the course) in order
    to provide the student an opportunity to
    experience the course instruction.
  • Length of the administration period is
    approximately the last 20-25 of the course.
  • Faculty are provided default SSI administration
    start and end dates, and are permitted to make
    minor adjustments to these dates within a
    predetermined timeframe. This ensures a more
    standardized and consistent data collection
    process.

33
Changing Administration Dates
  • Users can locate the course that they would like
    to change and select the Calendar icon

34
Changing Administration Dates
  • They then click on Next Month on the upper
    right hand side of the calendar to select the
    next month. The sample calendar above shows the
    course ends on 12/18 and the SSI closes on 12/11.

35
Changing Administration Dates
  • To extend the SSI to remain open until 12/18
  • Select 12/18. It will turn blue through 12/18.
    This changes in real-time.
  • Should you make a mistake, simply click back on
    the original survey end date of 12/11 or select
    another.
  • The latest you can extend the survey to is 12/24.

36
Response Rates
  • Student response rates are calculated and
    compared each term by program (academic org),
    campus, form type, and overall college.

37
Response Rates
    Submission Entries by Campus for 0400 Term Submission Entries by Campus for 0400 Term Submission Entries by Campus for 0400 Term Submission Entries by Campus for 0400 Term Submission Entries by Campus for 0405 Term Submission Entries by Campus for 0405 Term Submission Entries by Campus for 0405 Term Submission Entries by Campus for 0405 Term   SSI Submissions Split by Campus 0410 Term SSI Submissions Split by Campus 0410 Term SSI Submissions Split by Campus 0410 Term SSI Submissions Split by Campus 0410 Term
Campus   Roster Entries Opt-outs Average Roster Entries Opt-outs Average   Roster Entries Opt-outs Average
Allstate Center   1811 1388 0 76.6 2540 1537 23 60.5   1226 624 25 50.9
Clearwater Campus   16001 7194 0 45.0 15292 5469 48 35.8   6527 2209 35 33.8
Electronic Campus   9563 5937 0 62.1 10658 4785 100 44.9   5679 2674 57 47.1
EpiCenter   908 451 0 49.7 1022 295 1 28.9   271 113 2 41.7
Health Education Center   8484 5121 0 60.4 9271 4438 74 47.9   4242 2091 97 49.3
Open Campus   654 151 0 23.1 556 132 3 23.7   52 24 0 46.2
Seminole Campus   8461 4911 0 58.0 8101 3610 58 44.6   4226 1942 41 46.0
St Pete/Gibbs Campus   20991 9332 0 44.5 20706 7786 71 37.6   8336 2739 23 32.9
Tarpon Springs Campus   15736 7034 0 44.7 12634 6006 51 47.5   4691 2310 48 49.2
Totals   82609 41519 0 50.3 80780 34058 429 42.2   35250 14726 328 41.8
Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term Opt-outs are included in the number of Entries and were not available during Fall Term
38
Response Rate Comparison
  • Comparison with another Florida public community
    college
  • Another community college debuted its online
    course evaluation system in Fall 2007. Until the
    Spring 2007 term, this community college relied
    on a paper-based course evaluation system for its
    traditional classroom courses.
  • Response ratesPilot testSpring 2007 30
    classroom, 26 distance learningCollegewideFall
    2007 20 classroom, 26 distance learningSpring
    2008 24 classroom, 29 distance learning

39
SSI Revision Committee
  • An SSI Revision Committee was established in the
    Summer of 2009 to review the current SSI forms
    and make suggestions for revisions to the current
    items sets.
  • The new modality-neutral form was designed to
    address faculty and student engagement in the
    learning process and is being administered this
    term as a college-wide pilot.

40
Increasing Response Rates
  • While not the primary focus of the SSI Revision
    Committee, there was considerable interest in
    identifying and implementing ideas and processes
    to increase the response rate of the SSI.
  • Through creative brainstorming at the meetings
    and online, a list of best practices was
    developed.

41
Increasing Response Rates
  • SSI Revision Committee Recommendations
  • Send Constant Reminders (inescapable messaging)
  • Stress Confidentiality
  • Imbed link in announcement page -- inescapable
    messaging
  • Move every week to top of unit
  • Link to content graded or non-graded
  • Bring students to lab /use of lab
  • Give extra credit
  • Genuinely convey the message that the information
    will be used to make this class better
  • Hold activities on campus, e.g., a table
    highlighting importance for accreditation 

42
Increasing Response Rates
  • There was some concern that item 5 (linking the
    SSI to accessing exams or other course materials)
    was not acceptable from a legal standpoint.
  • The question was posed to the Office of General
    Counsel, and, in brief, the answer was yes, the
    faculty could block access to course materials
    assuming that the content blocked is just the
    next module or assignment the student would see,
    not specifically an exam.
  • The Office of General Counsel also advised
    against giving extra credit for taking the SSI.

43
Next Steps
  • Work with programs to improve response rates
  • Finish creating forms for programs using clinical
    items
  • Develop reports which provide trend information
    across terms and years
  • Finalize new items

44
Questions
45
  • Florida Association of Institutional Research
  • 2010 Annual Conference
  • Transiting to an Online Course Evaluation Model
    The Online Student Survey of Instruction

February 2010
About PowerShow.com