Comparing the Effectiveness of Alternative Approaches for Displaying EditError Messages in Web Forms

1
Comparing the Effectiveness of Alternative
Approaches for Displaying Edit-Error Messages in
Web Forms

• Bill Mockovak
• Office of Survey Methods Research
• Bureau of Labor Statistics

2
What is an edit message?

• Automated message presented to a user.
• The message may point out something wrong with an
  entry or ask the user to check or explain an
  entry.
• Display of the message could be automatic or
  under the control of the user.
• Hard vs soft

3
(No Transcript)
4
(No Transcript)
5
Why bother with this study?
6
Usability testing showed that users did not see
(missed) edit messages
7
And even when they saw them, some users did not,
or were not able to, follow instructions
8
High-Level Research Questions

• How big a problem is this?
• What factors are important?

9

• If a problem what can we do about it?

10
Basic Assumption

• Online edits will lead to higher quality data
  under the principle get it right at the source.

11
Characteristics of a Good Edit Message

• User sees and understands the message.
• The message points out
• Where the problem is (which item)
• What the problem is
• How to fix it

12
What factors might affect the usability of an
edit message?

• Hard or soft
• Position on screen/page timing
• User control (for initiating and closing message)
• Delivered One at a time vs. All at once
• In same or different window (pop-up)
• Visual characteristics (e.g., font size/type,
  color, layout, use of graphics)
• Complexity of message/readability/formatting
• Tone of message
• Scrolling page vs. page-by-page design

13
What was varied in this study?

• Location of message
• At top of page/screen, or
• Under item that triggered message
• Timing of message
• When user clicks Continue
• When user moves to next item in sequence

14
Three Approaches Were Used

• Approach 1. Top of page/screen, after all items
  on page were completed, user clicked Continue
• Approach 2. Under the item that triggered the
  edit, after all items on page completed, user
  clicked Continue.
• Approach 3. Under the item that triggered the
  edit, as soon as user moved to next item

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
Why study these design features?

• Includes an approach already used (Approach 1)
• Second approach (under item) is relatively easy
  to implement
• Third approach, more difficult to implement,
  programmers wanted experimental support to
  justify its use

19
What was kept constant in this experiment?

• Same visual design (and wording) of edit
  message.
• Message appears in same text box on same page.
• Same items/questions.

20
Survey Instrument Used

• Survey of Occupational Injuries Illnesses
  Web form mirrors paper closely
• Uses scrolling pages
• Three separate edits
• Total hours worked
• Date of injury
• Age of worker

21
(No Transcript)
22
Experimental Procedure

• Each user completed 3 scenarios
• In a single scenario, either one soft or two hard
  edits appeared
• Total hours worked soft edit, or
• Date of injury and Age of worker hard edits.
  
• Order of approaches and edits was counterbalanced

23
Procedure (continued)

• Since same edit could appear twice in one
  session, used different item values in scenarios
• Basic user task transfer data from paper form to
  Web form
• Talk aloud procedure used
• 42 paid participants, recruited by asking
• Experienced with Internet?
• Comfortable using keyboard mouse?

24
Procedural Glitches

• Total hours worked edit always triggered
• But
• Date of injury edit could be avoided
• Age of worker edit could be avoided on second
  appearance in same session

25
Key Variables

• Did the user notice the edit message on its first
  appearance?
• If noticed
• Was the proper corrective action taken on the
  first attempt? Also,
• User preference
• How did the approaches vary in terms of
  completion time?

26
What proportion missed the edit?
27
What proportion followed the directions? (saw
edit message followed instructions)
28
Overall Effectiveness of Edit(based on all
appearances of the edit)
29
Time to Advance to Next Page(Edit 1 the soft
edit)
30
Which Version Did Users Prefer?
31
User Ratings(where 10 is most positive)
32
Some Conclusions

• When a soft edit appears very early in a Web
  form, it can be frequently missed
• Varying the location and timing of edit messages
  did not have a significant effect (Caveats)
• As edit instructions become more complex,
  success decreases
• Detection of edit messages improves with
  increased experience with interface?

33
More Conclusions

• Rating scales are rough measures of usability
• Biased toward being overly positive
• Not a complete picture
• Observational data point out important problems
• Change Blindness can affect results

34
What is Change Blindness?

• The failure to detect what should be a very
  obvious visual change
• Very large changes can be made to a picture
  without observers noticing them
• Good experimental literature on this and how it
  occurs in a variety of situations
• http//nivea.psycho.univ-paris5.fr/Mudsplash/Natur
  e_Supp_Inf/Movies/Movie_List.html

35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
Theoretical Explanation

• Our eyes receive and send over 10 million signals
  to our brains each second
• The most liberal estimate is that people can
  process 40 pieces of information per second
• The rich visual environment we perceive is an
  illusion
• There is a major processing bottleneck

39
Theoretical background (continued)

• Change Blindness first noticed when change
  occurred during eye movement saccade, but effect
  is not limited to eye movements alone
• Necessary condition for Change Blindness
• Change occurs simultaneously with disruption in
  visual continuity
• Flicker effect

40
What happens in edit messages?

• Flicker effect
• Screen is displayed with survey questions
• Screen is refreshed with edit message now
  displayed, but some users fail to notice the
  change the edit message

41
What Can Be Done About It?

• Use a hard edit
• Place edit message on a separate screen
• Some other suggestions. Use
• Contrasting color
• Small, blinking change markers (to draw users
  attention)
• Results of this study suggest that
• Experience (general computer with interface) is
  important
• Scrolling page may contribute to effect
• (test page-by-page format?)

42
If I could repeat the study

• Get a better measure of user expertise with the
  Web and general computer skill
• Run more subjects
• Measure user literacy
• Higher literacy users tend to scan
• Lower literacy users tend to read word by word,
  take more time, and have a narrower field of view

43

• The End!

44
(No Transcript)
45
Some Other Links

• http//www.syntagm.co.uk/design/articles/cb.htm
• http//www.cs.bris.ac.uk/cater/PhD/ChangeBlindInf
  o/Examples.html