Zen and the Art of Web-Site Evaluation: Tools to assist in the successful development of an e-learning portal

1
Zen and the Art of Web-Site Evaluation Tools to
assist in the successful development of an
e-learning portal
Damian Gordon Ciarán O'Leary
DIT e-Learning Research Group
2
Introduction

• Develop an evaluation instrument for students to
  use to rate the quality of websites
• Using third-level computer science students with
  excellent information age skills
• Why ? To help identify appropriate web-based
  learning materials when building an eLearning
  portal

3
Defining Quality
4
Defining Quality

• Pirsig - Quality is a direct experience
  independent of and prior to intellectual
  abstractions.
• Kitchenham Quality is hard to define,
  impossible to measure, easy to recognise
• There are automated solutions, but we feel they
  are a quantitative approach to a qualitative
  problem

5

• What are some of the dangers associated with
  DHMO?
• Death due to accidental inhalation of DHMO, even
  in small quantities.
• Prolonged exposure to solid DHMO causes severe
  tissue damage.
• Excessive ingestion produces a number of
  unpleasant though not typically life-threatening
  side-effects.
• DHMO is a major component of acid rain.
• Gaseous DHMO can cause severe burns.
• Contributes to soil erosion.
• Leads to corrosion and oxidation of many metals.
• Exposure decreases effectiveness of automobile
  brakes.

6
Information Age Skills
Discernment
Searching
Monitoring
Just-In-Time Learning
7
Information Age Skills
Discernment
Searching
Monitoring
Just-In-Time Learning
The ability to know what you are looking for,
and to recognise it when you find it.
8
Information Age Skills
Searching
Monitoring
Just-In-Time Learning
Discernment
The ability to find what you are looking for.
9
Information Age Skills
Monitoring
Just-In-Time Learning
Discernment
Searching
The WWW is changing constantly, the nature of the
information you require changes constantly, so
searching for what you need to know is an ongoing
activity.
10
Information Age Skills
Just-In-Time Learning
Discernment
Searching
Monitoring
The quick, continuous learning and unlearning of
new skill sets
11
The Instrument
12
The Quest for Quality
Content
Web-Site
Technical Features
Authority
Form
13
Content
14
Content
15
Content

• Site Title
• Content Type (magazine article, tutorial,
  corporate web site etc.)
• Topic(s) Covered
• Covered comprehensively?
• Bibliographical References?
• Is the information practical?
• Language of Site (English, Finnish, etc)
• Does this site have information that other sites
  do not ?
• Is it an independent site (not too many out
  links)?
• Is it well cross-referenced (inter-linked)?

16
Content

• Site Title
• Search Engines attach a great deal of
  significance
• Titles shouldnt be too general
• e.g. instead of The WWW use
• A History of the World Wide Web from 1988 to
  2001

17
Content

• Content Type
• (magazine article, tutorial, corporate web site,
  etc.)
• Students are mostly interested in tutorial-type
  sites or sites with lecture notes on them
  (naturally)
• The right tools for the right job
• gt Government Web sites
• gt University Web Sites
• gt Corporate Web Sites

18
Content

• Topic(s) Covered
• Compare this with title of site, do they match ??
• Some students give table of contents of site

19
Content

• Covered comprehensively?
• Coverage
• How broad and deep is the coverage of a topic ?
• Does it give the what, who, when where, why and
  how of topic ?
• For new topics students are very interresting in
  finding site with good coverage of of a given
  topic
• Does it cover the entire history of the topic ?
• If it introduces new terminology, is it explained
  (or even better, linked to an explaination) ?

20
Content

• Bibliographical References?
• Scholarliness
• Scholarly articles always cite their sources (ot
  many pages cite their sources)
• The language of the site will be that of the
  discipline being covered
• Student arent looking for this
• sort of page, they want
• tutorial-type info and then
• assume it is scholarly.

NOTE Authors of quality information are more
inclined to publish in some other form, such as
books, journals, papers for recognition and/or
financial gain. Thus, websites must vie with each
other on the basis of being the most colourful
and dazzling, or most technically advanced
21
Content

• Is the information practical?
• This becomes the most important question once
  students are aware exactly what they are looking
  for
• Examples are sometimes too irrelevant to too
  complex
• It can often be very hard to find answers to
  specific questions

22
Content

• Language of Site (English, Finnish, etc)
• Nationality
• Cultural bias
• Variant spelling
• Variant terminology

23
Content

• Does this site have information that other sites
  do not ?
• Differentiation
• More Comprehensive, more up-to-date
• QUOTES
• Yes, it has more examples,
• better examples,
• takes a case study approach

24
Content

• Is it an independent site (not too many out
  links)?
• Coupling
• We distinguish between two kinds of sites, ones
  that provide links to other resources (portals)
  and those that provide original information
• A good site should contain all the necessary
  information and not rely on external sources
  which may become uncontactable.

25
Content

• Is it well cross-referenced (inter-linked)?
• Cohesion
• Is there a menu bar on each page ? Is there a
  site map ?
• Some websites require that you go back to the
  home page to get anywhere -(
• A good site should find an appropriate balance
  between inward-pointing and outward-pointing links

26
Form
27
Form
28
Form

• Is it intuitive to use?
• Is there one main concept per page (with at least
  5 pages)?
• Is the layout (and interface) consistent
  throughout the entire site?
• Is the grammar, punctuation and spelling correct?
• Is multi-media being used to assist learning?
• Does the site present the information in ways
  that other sites do not?

29
Form

• Is it intuitive to use?
• Navigability / Ease-of-Use / Workability /
  Conviviality
• Qualitive assessment of site
• To help avoid information overload always
  provide an index
• The simpler, the better. UI design, HCI design,
  Gestalt psychology

30
Form

• Is there one main concept per page (with at least
  5 pages)?
• Modularity
• Easier to design
• Easier to change a page
• Easier to find what you are looking for

31
Site Flowchart
HomePage
Section1
Section2
Section3

Page1
Page1
Page1
Page2
Page2
Page2
Page3
Page3



32
Form

• Is the layout (and interface) consistent
  throughout the entire site?
• Navigability / Ease-of-Use / Workability /
  Conviviality
• Usually this is not the case, different types of
  pages may require different layouts, e.g.
  tutorials vs. assessments
• Stylesheets can help keep things looking similar
• QUOTES
• Yes, makes it easier to use and find appropriate
  information
• the layout is confusing, as it is different for
  each topic

33
Form

• Is the grammar, punctuation and spelling correct?
• Accuracy
• International influence regional spelling
  suggests regional terminology
• Very detrimental to a sites credibility if not

34
Form

• Is multi-media being used to assist learning?
• Interactivity
• Help us determines the teaching style of the
  web-site (purely instructional, behaviourist or
  constructivist)
• In this survey, most web-pages that the students
  evaluated were principally text-based some had
  applets or JavaScript for illustrating
  programming concepts, but students are mainly
  interested in getting the information in a text
  based format. Could be their vocabulary of the
  internet is mainly text-based and younger
  students will favour most audio/video based
  learning on the web.

35
Form

• Does the site present the information in ways
  that other sites do not?
• Differentiation Appeal
• QUOTES
• Yes, its more informal
• Use of examples is great
• The layout is spacious
• teaching good website design by show bad
  websites
• Exercises at the end of each page
• Yes, boringly !

36
Technical Features
37
Technical Features
38
Technical Features

• URL of Site
• Does the site work on all browsers ?
• Does it require any additional downloads ?
• Is the site accessible ? Are there text/audio
  alternatives ?
• Do the graphics take a noticeable time to load ?
• Are all downloadables labelled with their sizes
  (e.g. Video 121Mbs) ?
• Does the site provide a search function ?
• Does the site provide additional technical
  features that other sites do not (e.g. Bulletin
  Board, mailing lists, search facility)

39
Technical Features

• URL of Site
• Visibility
• Search Engines are reducing the importance of
  URLs
• Top-Level Domains
• .com .edu .gov .net .org .tv
• .ie .uk .us .au .sa

• Jakob Nielsen
• A URL should by easy-to-remember, easy-to-type
  and easy-to-spell
• URLs that visualise the site structure
• URLs that are "hackable" to allow users to move
  to higher levels of the information architecture
  by hacking off the end of the URL
• persistent URLs that don't change

40
Technical Features

• Does the site work on all browsers ?
• Compatibility
• Does it work on Internet Explorer ? Netscape
  Navigator ? Mozilla ? Opera ? Cello ? Emacs ?
  K-Meleon ?
• What about backwards compatibility ? Older
  versions of browers ?
• Does it work for text-based browsers ? Lynx ? w3m
  ? EnterWEB ? -- consider the emerging economies
• Problem include frames, Javascript,
  appletsFrontPage.

41
Technical Features

• Does the site work on all browsers ?
• Spot the Difference ...

Internet Explorer
Netscape Navigator
42
Technical Features

• Does it require any additional downloads ?
• Compatibility
• e.g. Java classes, Flash, Quicktime, activeX
  components, cosmo player.
• Students may not be allowed to install software
  on school computers, so should try to avoid an
  excess of technical wizardry.

43
Technical Features

• Is the site accessible ? Are there text/audio
  alternatives ?
• Accessibility
• Very important -- many of the students surveyed
  were not farmiliar with the term accessibility
  -(
• The W3C (World-wide Web Consortium) has a body
  that coordinates this
• WAI - Web Accessibility Initiative

44
Technical Features

• Do the graphics take a noticeable time to load ?
• Retrievability

• Where did Boo go wrong?
• It demanded users have the Flash plug-in,
• then forced them to navigate pages of animation
  to get to the place where they could order
  something.
• It hid the navigation under cute graphics.
• It crashed browsers.
• It launched new windows at every opportunity.
• It demanded a fast connection (in theory 56k, but
  higher in reality).
• It blocked Mac users entirely

Some students feel that sites with no graphics
are boringyou cant please everyone
45
Technical Features

• Are all downloadables labelled with their sizes
  (e.g. Video 121Mbs)
• Accessibility
• Shows a consideration for your audience

46
Technical Features

• Does the site provide a search function ?
• Searching
• Again, shows a degree of consideration for your
  audience
• Allows students to find information quickly

47
Technical Features

• Does the site provide additional technical
  features
• (e.g. Bulletin Board, mailing lists, search
  facility)
• Differentiation Instruction Style
• Is the site purely instructional, do it have
  regular quizzes (behaviouristic) ? Interactive
  tools (Const) ?
• HTML tutorial should have HTML validator
• Tests, Demos, Notice boards, Discussion boards,
  Mailing lists, Drop down lists, Help files, User
  formatting, Shopping carts

48
Authority
49
Authority
On the internet, no one knows I'm a dog.
50
Authority

• Is it clear who the author(s) are ?
• What organization do they represent ?
• Can you contact either the author(s) or the
  organization ?
• Is the organization or author(s) well-known ?
• Date of finding site
• Does the site have a date of creation and date of
  last update ?
• Is this site linked to from anywhere ? If so is
  the parent site a credible authority ?
• Are there a lot of out-links ? What sort of sites
  do they go to ?
• Is there additional authority information you can
  learn from the URL (e.g. the page is part of a
  users site, it may be a temporary site, it may
  be a free site) ?

51
Authority

• Is it clear who the author(s) are ?
• Objectivity Scholarliness
• Are there any political, ideological or other
  biases ?
• Is the researcher in the domain ?

52
Authority

• What organization do they represent ?
• Objectivity Scholarliness
• As before, are there any political, ideological
  or other biases ?
• Is the organization in the domain ?

53
Authority

• Can you contact either the author(s) or the
  organization ?
• Credibility
• Increases user confiedence

54
Authority

• Is the organization or author(s) well-known ?
• Credibility
• Increases user confiedence

55
Authority

• Date of finding site
• Currency
• For future reviews

56
Authority

• Does the site have a date of creation and date of
  last update ?
• Currency
• Students need to realise that if the author or
  sponsorship of the information is not clearly
  identified and if there is no indication of date
  of publication or update, then the materials
  should not be used.

57
Authority

• Are there a lot of out-links ?
• What sort of sites do they go to ?
• Credibility
• It is important that links are not to sites with
  excessive advertising, irrelevant materials or
  legally compromising material

58
Authority

• Is this site linked to from anywhere ? If so is
  the parent site a credible authority ?
• Credibility
• Again lends credibility, key factor in
  calculating importance of site in search engines

59
Authority

• Is there additional authority information you can
  learn from the URL (e.g. the page is part of a
  users site, it may be a temporary site, it may
  be a free site) ?
• Stability
• www.cal2003.com
• www.comp.dit.ie/dgordon
• http//www.ireland.com/newspaper/breaking/2003/040
  7/index.htm
• http//loginnet.passport.com/login.srf?id2svcma
  ilcbid24325msppjph1tw0fs1fsa1fsat12960
  00lc2057_langEN

60
Outcomes

• which features students like about websites
• identified and refined their information age
  skills
• which websites appeal to students
• a set of standards that can be applied when
  designing our own materials
• a corpus of websites that are rated, thus
  embodying Communal constructivism

61
Learning Styles
62
The Need for Standards

• SCORM / AICC / IMG ...

• XML defined Websites

• META tagged HTML sites

• Standard HTML Websites