Evaluating

1
Evaluating Choosing Instructional Software
Edited by Sandra Lebrón-Lozada, M. Ed. May,
2006
2
IntroductionObjectives and Expectations
3
Workshop Objectives

• By the end of this workshop you should
• Know the distinctions among software applications
  used in instruction.
• Be aware of pedagogical issues regarding the
  effectiveness of instructional software in
  different disciplines or content areas.
• Overview the evaluation process including the
  development of evaluation criteria, the selection
  of an evaluation instrument, how to analyze
  evaluation results, and who needs to be involved.

4
Lesson OneGood Reasons to Use Software
5
Improve upon ExistingGoals and Outcomes

• Learning Outcomes broad goals focusing on
  course goals and core competencies in a course or
  curriculum for example improving critical
  thinking skills.
• Learning Bottlenecks specific problem areas
  that are difficult to learn and if not mastered
  can hinder progress in a course for example
  mastering fundamental math concepts.

A Taxonomy of Goals, Ehrmann, 2002
6
Establish New Goals that Depend upon Technology

• Computer Literacy Basic knowledge of computer
  use for communication, authoring and information
• Web Research The ability to find and evaluate
  information from web-based sources
• Learning Online How to communicate and work
  effectively in asynchronous, networked
  environments

A Taxonomy of Goals, Ehrmann, 2002
7
Activity 1 Finding Good Reasons

• Sketch out your thoughts on the following
  questions
• What are the current ways in which software is
  used in your course (s)?
• Have these uses improved instruction over more
  traditional methods?
• What new course goals or learning bottlenecks
  could benefit from the use of technology?

8
Value Defined

• Software is valuable if it has the capability
  of being used to help improve teaching and
  learning. As the cases show, indicators of value
  can include evaluation results, awards won,
  testimonials from users, and the like. Notice
  that our definition of software's value is
  independent of the extent to which it is used.

On Value, Viability and Success, Ehrmann, 1994
9
Viability Defined

• Software is viable if it is used by enough
  people for a long enough period of time that all
  its investors (original developers, fonders,
  publishers, institutional support staff, faculty,
  and students) can justly feel that they each have
  received an adequate return on their own
  investments in developing, acquiring, and/or
  learning to use the software.

On Value, Viability and Success, Ehrmann, 1994
10
Some Finds on Viability

• From studies published in Valuable, Viable
  Software in Education, funded through the
  Annenberg/CPB Project
• If computer software is not adequately upgraded
  to cope with changes in operating systems, it
  probably will not last long enough to achieve
  viability.
• most of the valuable viable softwarefell into
  the category of worldware

On Value, Viability and Success, Ehrmann, 1994
11
A Typology forInstructional Software

• Courseware
• Learning Management System (LMS)
• Course Management System (CMS)
• Open Source Content Management Systems (OS/CMS)
• Worldware
• Low Threshold Applications (LTA)

12
A Broad Definition

• Instructional software can be defined
• as any software used for instruction.

13
1. Courseware- Educational Software
Made for the teaching and learning of specific
content. Ex. Math, Science, Social Sciences,
English, etc.
14
Definition of Courseware

• Software designed specifically for
• use in any educational setting classroom,
  computer labs, network, and web-based that can be
  access from different devices

15
Virtual Laboratories
16
Virtual Laboratories Lab View
17
Virtual Laboratories Quiz View
18
DiscoverEcon
19
DiscoverEcon Chapter View
20
DiscoverEcon Exercise View
21
Learning Management Systems (LMS) and Course
Management Systems (CMS)
Software that is used to design, deliver and
manage an online course.
22
2. Learning Management Systems (LMS)

• Learning management systems (LMS) refers to
  software that primarily acts as an electronic
  registrar by electronically performing various
  enrollment and related tasks.
• LMSs were originally designed for workplace
  learning environments, and specifically perform
  some or all of the following tasks
• registration
• track participation (classroom attendance,
  sign-on of online courses)
• track of completions (including final scores or
  grades)
• testing
• follow-up discussions with participants
• aggregated reports, such as the number of people
  registered for particular courses
• transfer of information to other systems, such as
  human resource information systems
• process charges for courses, such as tuition
  payments and transfer payments among departments
• course catalog
• skills management.

http//www.learningcircuits.org/2005/nov2005/carli
ner.htm
23
Mymathlab
To login as an instructor to preview an existing
HCC course go to http//www.coursecompass.com/cci
ndex.html Login Name AWMML27 Password AWMML27
DistanceLearning.org Glossary http//www.distancel
earning.org/glossary/
24
3. Course Management Systems (CMS)

• Course management systems (CMSs) are online
  systems that were originally designed to support
  classroom learning in academic settings, such as
  universities and high schools. CMSs provide
  instructors with the ability to perform the
  following tasks
• Place course materials online. Most CMSs provide
  pre-programmed buttons for the course syllabus,
  course schedule, and course materials linked to
  specific lessons, such as copies of readings and
  PowerPoint slides from lectures.
• Track student progress through assessment
  features, which enable instructors to give
  quizzes and tests online, and an online
  gradebook, where instructors can post student
  grades.
• Discussion board, where instructors and students
  can discuss readings and continue class
  discussions between formal class sessions.

25
3. Course Management Systems (CMS)

• Other communications tools, which let instructors
  send announcements to classes and communicate
  individually with students
• Lock box for students, where students can store
  class materials in a safe placeeither a
  presentation to give later in class or backing up
  class assignments in a safe place.
• Course statistics, which provide information on
  the use of the course site, including who used
  the course site and when.

http//www.learningcircuits.org/2005/nov2005/carli
ner.htm
26
WebCT Course Management _at_HCC
http//webct.hccs.edu/
27
WebCT Course Management
28
WebCT Tools
http//www.colostate.edu/webct/webct_tools.htm
29
WebCT with e-Pack
http//www.blackboard.com/extend/Digital_Content/I
nstructors.htm
30
More about epacks

• ..\WebCTepacks\wbct-texas2004-epacks.htm

31
4. Open Source/Course Management System (OS/CMS)
Free software that is used to design,
deliver and manage an online course using open
sources available on the Internet
32
Open Source/Course Management System (OS/CMS)
Benefits? http//www.openknowledge.org/writing/ope
n-source/scb/why-open-source.html http//open-sou
rce.gbdirect.co.uk/migration/benefit.html Myth? h
ttp//www.cio.com/archive/030104/open.html
33
Open Source/Course Management System (OS/CMS)
Free software that is used to design,
deliver and manage an online course using open
sources available on the Internet
34
5. Worldware
Software that is developed for purposes other
than instruction but is also used for teaching
and learning.
35
Benefits

• Widely available
• Have a real-world application
• Easy to correlate to curriculum

Low Threshold Applications (LTAs), TLT Group
36
A Sampling of Worldware

• Word processors
• Spreadsheets, statistical and other mathematical
  software
• Software for managing library services
• Communications software
• Tools for design and composition in fields as
  diverse as architecture, desktop publishing and
  music
• Tools for modeling and simulation
• Research databases online and on discs
• Programming languages

On Value, Viability and Success, Ehrmann, 1994
37
6. Low Threshold Applications - LTA
Software that is reliable, accessible, easy to
learn, non-intimidating and (incrementally)
inexpensive.
Low Threshold Applications (LTAs), TLT Group
38
LTA Benefits

• Ubiquitous
• Reliable
• Easy to Get Started
• Offer Immediate Results
• Precipitate Long Term Changes

Low Threshold Applications (LTAs), TLT Group
39
Possible LTAs

• Communication Software (Email / Listserv /
  Chat / Discussion Boards)
• Web Software (Browser, Search Engine)
• Productivity Software (Word Processor,
  Spreadsheet, Presentation)
• Collaborative Software (Wikis, Weblogs)

40
Activity 3 Consider all options

• Sketch out your thoughts on the following
  questions
• Based on your notes from Activities 1 , what
  software options do you think are most viable for
  your course(s). Keep in mind your course goals
  and desired learning outcomes.

41
The Next StepThe Software Evaluation Process
Using Instructional Design Principles Analyze/cont
ent, instruction, typology, and
resources Design/ Evaluation Plan and
Timeline Develop/Criteria and Eval.
Instrument Implement/ Conduct Assessment Evaluate/
Results and Recommendations
42
Sample Standard Criteria

• The ERMES (EuRopean Multimedia Educational
  Software network approach
• http//acm.org/ubiquity/views/g_avellis_1.html
• Educational features of the software
• Technical features
• Ease of use (usability)
• Content
• http//www.ifets.info/journals/5_2/avellis.html
• the content which is to be taught
• the delivery media used to provide information
• the user interface, that is the way the
  educational software presents itself to the user
  
• the interaction devices, by which the user
  interacts with the computer, making choices,
  answering questions or performing activities, and
  is provided with feedback to each response and,
• the instructional strategy adopted.  

43
Evaluation Review Criteria- various

• http//www.evalutech.sreb.org/criteria/index.asp

44
Selecting Evaluation Instruments

• Heuristic Approach for evaluating
• Interface design heuristics after Nielsen
• Educational design heuristics after Quinn
• Content design
• http//www.usq.edu.au/users/albion/papers/ascilite
  99.html

45

• Rating Scales Surveys
• http//www.ed.brocku.ca/jkerr/sftwreva.htm
• Teaching, Learning, and Technology Group.
  http//www.tltgroup.org

46
Rubricsfor Evaluating Instructional Software

• WebCT Exemplary Rubric http//www.webct.com/exem
  plary/
• Texas Higher Education Coordinating Board guide
  for Incorporating the Principles of Good Practice
  into electronically-based Courses
  http//distance.hccs.edu/links/fac_res.htm
• http//www.arches.uga.edu/eshealy/rubric.htm
• http//www.ncsu.edu/midlink/sw.eval.rubric2.doc
• http//hagar.up.ac.za/catts/learner/eel/Conc/conce
  ot.htm

47
OS/CMS Assessment Evaluation Process Adapted
from Univ. of Northern Iowa http//fp.uni.edu/its/
et/lms/cms_eval.pdf

• Needs Assessment
• What are we doing this? Determine the purpose.
• Agree on the evaluation methodology and process.
• Identify evaluators
• Identify resources
• Search and select evaluation method, criteria
  and process
• Gather information on products/vendors
• Determine any costs
• Create timeline

48
OS/CMS Assessment Evaluation Process cont.
Adapted from Univ. of Northern Iowa
http//fp.uni.edu/its/et/lms/cms_eval.pdf

• Evaluation Process
• Identify products from selected list
• Choose features - http//www.edutools.info
• Assign ranks (1-5) for weighting as criteria
  to create a short list of products
• Choose criteria for evaluating the features- i.e.
  rubric w/scores scores weight
• Perform individual or group evaluations in
  three areas
• Learner Tools
• Support Tools
• Technical Specifications
• Report findings and make recommendations

49
The Pilot Test

• Who should be involved to make your pilot a
  success? All stateholders.
• Students
• Faculty and Colleagues
• Technologists
• Administration

50
Students

• Have your students actively participate by
• Giving first impressions about the
  softwaresease of use, accessibility, and its
  currency
• Adding an assessment of the software to the
  evaluation of the course

51
Faculty/Colleagues

• Invite your colleagues to
• Start a users group by finding others who might
  be interested in using the software
• Give a second opinion have them sit in on a
  class session where the software is used and give
  you their impressions
• Provide the occasional sanity check are you
  still focused on your original goals or has the
  software taken over?

52
Technologists

• Have your technologists answer these questions
• What are the technical specifications and
  hardware requirements for using this software?
• Is there adequate user documentation for the
  product? Is there an active user or developer
  community?
• Will there be participation by technology staff
  during the evaluation? What about ongoing
  technical support?

53
Administrators

• Involve chairs and administrators to
• Get ideas for extra funding from grants,
  stipends, and other outside sources
• Set realistic deadlines to meet and evaluate the
  progress of the pilot
• Make a long-term commitment to fund the software
  based on agreed upon outcomes

54
Production Credits

• Sandra Lebron-Lozada, Instructional Design
  Coordinator
• HCC-SEC
• Jordan Carswell, Web Graphic Designer
• HCCS-NWC

55
Have a great day!