What does getting WET (Web Event-Logging Tool) Mean for Web Usability?

1
What does getting WET(Web Event-Logging
Tool)Mean for Web Usability?

• Michael P. Etgen Judy Cantor
• User Experience Engineering Division
• ATT Labs

2
Web Data Collection Approaches

• Traditional Usability Data Collection
• Hand coding in person or from video
• Advantages ask questions, get subjective
  responses, see exactly what happened
• Disadvantages expensive, time-consuming, can
  introduce human error, may require extra
  equipment

3
Web Data Collection Approaches

• Server Logs
• Servers automatically log user requests and
  server responses
• Advantages automatic and inexpensive
• Disadvantages
• no data collection on Browser buttons/menus
• cant log client-only interactions (e.g.,
  anything that doesnt send a request to the
  server)
• validity of data - dynamic IP addressing and
  browser/proxy caching can provide misleading
  information
• huge amount of data to parse

4
Web Data Collection Approaches

• Automated Data Collection Software
• Software runs in addition to the browser, or
  logging browser developed
• Advantages automatically collects wide range of
  data at Operating System/Application level
• Disadvantages
• cost of buying or developing software
• installation of software on each users client,
  and retrieval of data from each client
• platform dependence
• cant get content-specific data

5
Web Data Collection Approaches

• WebVIP from NIST
• Copy site, add inline event handling code to each
  link
• Advantages client platform independence, data is
  content-specific, inexpensive
• Disadvantages
• requires copying entire site, and altering code
  for each link
• no data collection on Browser buttons/menus (use
  inferred)
• wont work if Javascript is disabled
• no data collection on HTML elements other than
  links (e.g., checkboxes, radio buttons, selection
  lists, etc.)

6
Web Data Collection Approaches

• An Alternative Web Event-Logging

(1) Browser generates events in response to user
interactions with web page click, load, change,
mousedown, etc.
(2) Javascript can temporarily log event time,
type, and source info on Client
(3) Javascript can format event log and send it
back to server
(2)
server responses
(1)
Internet
client requests
(3)
Server Side
Client Side
7
Web Data Collection Approaches

• Web Event-Logging Tool
• Advantages
• content-specific, client-side data collection
• client / server platform independence
• code is modular, so easy to insert/delete, does
  not require copied site or adding code to
  individual HTML elements
• you choose events/pages/features/etc. to log, and
  only log those
• inexpensive
• Disadvantages
• wont work if Javascript is disabled
• no data collection on Browser buttons/menus (use
  inferred)
• Solution for getting data back is specific to
  site/application

8
Web Data Collection Approaches

• Web Event-Logging Tool
• Examples of events that can be logged and the
  sources of the event
• load, unload, move, resize - window
• click - link, area, button, radio, checkbox,
  reset, submit
• mousedown, mouseup - link, button, radio,
  checkbox, reset, submit
• change - text, textarea, select
• keypress - text, textarea, password
• mouseover, mouseout - link, area
• blur, focus - window, text, textarea, password,
  select

9
Implementation in a Web App

• Background Planning and Coordination tool for
  Executive Council
• Functionality Meetings, Action Items, Meeting
  Minutes Presentations, Confirm Availability,
  Contact Members
• Technology ColdFusion and MSAccess database

10
Implementation in a Web App

• Configuring WET
• Basic Steps for Any Site/Application
• Specify events and temporary storage method
• Place a copy of WET.js on web server
• Insert call to WET.js in all document headers
• Specify method for log retrieval
• Additional Steps for current trial
• Code for turning WET on/off based upon user login
• Code for Start/Stop links, created as dynamically
  positioned layer so wont scroll off screen

11
Implementation in a Web App
Application Before WET
12
Implementation in a Web App
After WET is configured and running
13
Implementation in a Web App

• Method
• Study directions, task list, and background
  information were sent to participants by e-mail
• Testing was remote, no direct contact between
  testers and participants while performing tasks
• Four participants completed trial
• 7 Tasks covered all the basic functionality of
  the application

14
Implementation in a Web App

• Method
• Metrics and Objectives
• Three metrics were calculated for each task and
  compared to usability objectives
• Task Completion - examined page access
  information in log
• Time to Complete Task - subtracted time for click
  on stop link from click on start link
• Deviation from Ideal Interaction Model -
  comparison to expert path

15
Implementation in a Web App

• Example from Trial Tasks
• Contact a specific council member (e.g., Judy
  Cantor) and send an e-mail to them indicating
  that you are finished with the trial.
• Metrics and Objectives
• Task Completion - 3 out of 4 successfully
  complete task
• Time to Complete Task - 3 out of 4 complete task
  within 120 seconds.
• Deviation from Ideal Interaction Model - 3 out of
  4 deviate from ideal no more than twice

16
124644.242, load, home.cfm 124649.726, click,
Linkjavascripttask(start), home.cfm 124654.8
67, click, Linkcontact.cfm, home.cfm
17
124657.570, load, contact.cfm 124704.258,
click, checkboxusermike, contact.cfm 124707.89
8, click, submitCompose Mail, contact.cfm
18
124708.103, load, mail_form.cfm 124710.346,
change, textsubject, mail_form.cfm 124717.582,
change, textmessage, mail_form.cfm 124735.957,
click, submitsend, mail_form.cfm
19
124739.586, load, contact.cfm 124742.961,
click, Linkjavascripttask(stop), contact.cfm
20
Implementation in a Web App

• Results for Example E-mail Task
• Task Completion 4 out of 4 successfully
  completed the task
• Time to Complete Task 3 out of 4 successfully
  completed the task in less than 120 seconds.
• Deviation from Ideal Interaction Model 4 out of
  4 had no more than 2 deviations

21
Implementation in a Web App

• Lessons Learned
• Showing companion start/stop graphics was not
  reliable method for indicating task start
• participants couldnt recall if theyd clicked
  start, so we had to use first click on menu
  item on home page.
• all remembered to clickstop when finished
  though.
• Possible solutions
• Place start in dialog window on top of home
  page so the user cannot proceed without clicking
  it.
• Change appearance of start after it has been
  clicked

22
Implementation in a Web App

• Lessons Learned, continued
• Think about what needs to be logged carefully
• Amount of data can be overwhelming
• For Usability Testing choose the minimum number
  of events that reflect how site is being used
• For Usability Monitoring log only certain pages,
  or certain functionality
• Think about temporary storage method in terms of
  site/application architecture
• some possibilities are session-only cookies,
  frame variables, variables in parent window, etc.
• best to use method which least alters
  site/application

23
Future Directions

• To Do List
• Apply the technique to other web-relevant
  technologies that use events (e.g., Java
  applets/applications), and integrate that data
  with the Javascript WET data
• Investigate whether generalized configuration
  modules are possible for certain types of
  sites/applications, so implementation isnt as
  site/application specific
• Make code publicly available for free to anyone
  who wants to use it

24
Future Directions

• Wish List
• If we can create generalized configuration
  modules, then we could create a wizard for
  installation and configuration
• For a basic set of metrics, create an analysis
  wizard for data summarization and analysis

25
Conclusion

• The technique of Web Event-Logging (as embodied
  in WET) provides the usability engineer with a
  method of collecting web usability data that is
  automated, customizable, and comprehensive.
• What does getting WET (Web Event-Logging Tool)
  Mean for Web Usability?
• It means you can gather usability data on a
  site/application within the limited time usually
  afforded to such activities on web projects.