User Experience Research for Open Software Developers

1
User Experience Researchfor Open Software
Developers

• OReilly Open Source Convention
• July 23, 845AM-1215PM

2
The Schedule

• Who we are
• Introduction
• Open Source Software and the User Experience
• User Experience Research Techniques
• Overview
• Usability Testing
• Task Analysis
• Feedback Analysis
• Log File Analysis
• Questions

3
Introduction

• Mike
• HotWired
• Put together OpenOffice research plan
• Wrote a book in StarOffice
• Wrote a piece for sendmail.net on OSS usability
  in 1999 that got some play ("it's the user,
  stupid)
• Lane
• Early DejaNews UI designer
• Technical editor for OReilly Web and Interface
  books
• You all
• What brings you here?

4
Open Source Software and the User
ExperienceAssumptions

• Open Source
• We all know what Open Software is
• Some OSS products have great interfacesfor their
  target audience (Apache, perl)
• This subject of this tutorial
• Software as a means, not an end
• Software thats designed to be used by users who
  are not programmers
• GUI, not CLI
• All the answers cannot be found in rules and
  guidelines. Each piece of software is different.

5
OSS and the UE Problem Overview

1. There arent enough Open Source UI Designers
2. Usability can be difficult and painful
3. Pandemic featuritis
4. Copying without understanding is not enough
5. Design is tough to do by committee
6. Running does not equal working
7. OSS Software is not written for specific audiences

6
1. Not Enough OSS UI Designers, why?

• Designers cant code
• but coders can place interface widgets
• Designers cant make software by themselves
• Cultural gap
• Different academic traditions
• Design is harder to share than code
• Seen as window dressing by some

7
2. Good usability can be painful

• Many of the little details which improve the
  interface are not exciting or satisfying to work
  on, so they get fixed slowly (if at all).-
  Matthew Thomas, Mozilla UI designer
• Good UE design requires great attention to detail
• Solutions are uncertain and often compromises
• You cant kludge it

8
3. Pandemic Featuritis

• EMACS is the granddaddy of OSS bloatware
• but Mozilla is the reigning monarch
• Everyone has a favorite feature and wants their
  15 pixels of fame
• In the name of flexibility
• every idea becomes a feature
• every feature has options
• every option is made a preference
• The ultimate expression of this is the concept of
  skins, but skins fix nothing

9

10
4. Copying doesnt fix anything, either

• Just because Apple did it doesnt mean its
  right.

iTunes
Rhythmbox
11
Copying is not enough

• Surface issues
• MS and Apple make plenty of bad UI decisions
• Example
• Deeper issues
• iTunes may have a good interface, but WHY?
• Good interface design is an expression of good
  architectural design decisions

12
Copying without understanding is like telephone

• What is he holding?
• Fan blade?
• Cinnamon stick?
• Sausage?

13
Copying without understanding amplifies mistakes
1567
1750
Today
Taken from http//www.cs.man.ac.uk/playing-cards/t
mfaq/spades.html
14
5. Running does not equal working

• Just because it compiles doesn't mean it works
• Just because it works doesn't mean it works well
• Just because it works for you doesn't mean that
  it works for everybody
• The default matters what you give most users is
  what most users will usually use

15
6. OS Software is not written for specific
audiences

• You are not your audience!
• You do not
• see things like they do
• know what they know
• want what they want
• work how they work
• This is critical information when designing
  software interaction

So how do you figure out all of these things?
16
User Experience (UE) Research!

• The study of what makes peoples lives difficult
  and how to make them easier
• AbilitiesWhat they can understand and do
• NeedsWhat people need to make their life easier

17
Abilities are discovered through Capability
Research

• Timing After functionality has been implemented
• Purpose Investigates physical/psychological/memor
  y abilities, reactions
• Desired Outcome An understanding of which design
  solutions are usable by the desired audience
• When a problem has been identified and a solution
  created thats designed to satisfy the audiences
  desires, capability research determines whether
  they can actually use it in the manner intended

18
Needs are discovered through Conceptual Research

• Timing At the beginning of the design process
• Purpose Define your audience, understand problem
  scope, investigate peoples needs and current
  methods related to the task your software is
  trying to enable
• Desired Outcome A foundation for developing a
  product (and therefore the interface to your
  product) based on how your users actually work
• Conceptual research assumes a good interface
  comes from understanding what your entire user
  base needs
• It finds ways to create solutions that fit in
  with peoples existing processes and solve
  specific problems theyre having

19
Research methods should work together

• Its possible to create a product that works
  well, but solves no problems (The Toilet Cozy)
• Its possible to create a product that solves
  problems, but people cant use it (The
  Three-Pedal Bicycle)

20
How is UE Research related to Interface Design?

• UE research is often lumped in with UI design
• Its not the same thing
• UE Research defines problems
• Interface design, like programming, finds
  solutions
• The better the problem definition, the clearer
  the path to the solution
• (Tangentially interface design is not making
  things pretty, thats graphic design -)

21
User Research in Open Source Development

• UE Research and User Experience development in
  Open Source software is in its infancy
• The OSS model is much more geared toward
  development than design
• There are UI folks who love to design like you
  like to code, and you should encourage and
  welcome them into the community
• Open question How can designers participate
  without submitting patches or waging wars in
  Bugzilla?
• Because there are many more OSS developers than
  designers, coders need to know about this stuff,
  too

22
UE Research in the Development Process The Ideal

• Highly iterative
• Many small steps, rather than a few giant ones
• Research at every possible step

23
User Research in the Development Process
Practical

• Developing an iterative approach to user research
  for a product takes time (especially when theres
  no existing place for it in the development
  process, as with most OSS)
• Bootstrapping an existing development process and
  team is required
• Linear process
• Do all research in two lumps conceptual and
  ability

24
Integrating User Research into Development

• Integrate research into all phases of development
• No matter what stage a product is in, there's
  some research you can do
• Make sure research is actionable
• Time the research for the needs of the product
  and the abilities of the development team
• Example Don't research button wording before you
  know whether the audience wants the buttons
  functionality
• Avoid research paralysis
• It's OK to make decisions without first asking
  people, just dont make all your decisions that
  way
• Dont get distracted by research and forget the
  product
• Be open-minded
• Users arent stupid, they just dont have as many
  reasons to care as you do

25
How to get started

• Dont worry about doing it right
• UE research scales well
• Its always worth getting some data
• Start simple friends and family

26
The underlying pattern of all UE research

• The basic process that nearly all UE research
  methods follow

• Define a test audience
• Find them
• Ask them some questions
• Look for similarities in their responses
• Generalize similarities into trends
• Respond to problems through design/development
• GOTO 2

27
Usability Testing
28
Usability Testing An Overview

• What is usability testing?
• Way of researching peoples abilities
• Uses structured interviews
• Watch people as they use your product
• When can you use usability testing? Any time!
• During initial design work
• Between design iterations
• To find out why your users complain so much
• Before a redesign

29
5-step Process

1. Define an audience and their goals
2. Create tasks based on those goals
3. Bring in some users that match the audience
   description
4. Have them try the tasks
5. Look for trends

30
1. Define an Audience and their Goals

• You are making a product for some reason.
• You have decided that some people in the world
  can make their lives better with your idea.
• These are the questions you need to answer
• Who are those people?
• Who will be the largest group of users?
• What differentiates them from everyone else?
• Age? Interests? Computer experience? Problems?
  
• What does your product do for them?
• Why are people going to use it?
• Why is it valuable to them?
• If you were at a loud party and had 30 seconds to
  describe your product to someone who had never
  heard of it, what would you say?

31
2. Create Tasks

• Write down the five most important functions of
  your product
• Not the features, the functions
• What problem is being solved?
• What are the five things that people should
  people be able to do above all others?
• Basic functions someone should be able to do with
  a word processor
• Enter text
• Edit text
• Format
• Search and replace
• Save
• Create a one to two sentence scenario for each
  function
• An example of someone using each function, framed
  as a question
• Written from their perspective

32
Example Mailing List Management Tasks

• You have been asked to create a mailing list for
  the California Furby Collector Club. Here is a
  list of the five people in the club. How would
  you go about creating this list?
• Your Furby list has been a great success and you
  would like to have a record of all of the great
  discussions that happened on the list. You have
  heard that you can set up a permanent collection
  of all of the mail that goes to your mailing
  list. How would you go about setting this up?
• One of your list members has decided to stop
  collecting Furbys and move on to AIBOs. How
  would you take him off of the list?

33
3. Find Some Representative Users

• Narrow down your intended audience ahead of time
• Pick the people who will give you the best
  response
• Often a subset of the largest group of users (the
  middle of the bell curve)
• People whose problems generalize to the largest
  number of other people if you solve the problem
  for them, you solve it for many other groups, too
• Be specific about who you want
• Dont recruit people who could have strong
  opinions about the product separate from the
  interface
• Some people know too much
• Market researchers
• Designers
• Software developers

34
Where to Find Users

• For simple research friends, family, coworkers
• Pros cheap, easy
• Cons bias (they may know too much), not close
  enough to the real target audience
• For more complex research use a recruiting
  agency
• Pros can get people who know nothing about the
  product, can get people who are exactly your
  audience, can recruit people in a variety of
  geographic locales
• Cons money

35
Where To Find Users (Cont.)

• Some mid-range options
• Existing user base, customer support inquiries,
  advertise on existing site
• User groups, email discussion lists
• Traditional market research means classified
  ads, etc.
• Start immediately the more time, the better the
  subjects, the better the outcome

36
Use a Screener

• A screener is a simple script to help you
  consistently select people to participate in your
  research
• 20 or so questions that narrow in on the group
  youre after
• Order questions from generic to specific
• Be very clear and specific
• Avoid jargon

37
Example Profile Mailing List Management Users

• Demographic Profile
• Any gender, but prefer roughly even split
• Any age
• Technographic Profile
• Email users
• Online at least 6 months
• 8 hours Net use per week, including email and
  Web surfing
• Don't consider themselves power users
• Have Internet access at work
• Behavior
• Have been on at least one mailing list for at
  least a month
• Have never managed a mailing list
• Do not work in marketing, or software development

38
4. Run the test!

• Create a comfortable space
• Set up a typical computer
• Nothing tricked-out
• User would expect to do it
• Ideally, users own computer
• Prepare the participant
• Explain what its all about
• Tell them that theyre evaluating the product and
  that any problems are not their fault
• Ask them to say all their thoughts aloud
• Describe the product using your 30-second party
  speech

39
Run the test! (cont)

• Give them the list of tasks presented as goals
• Give them exactly as much detail as they need to
  get the job done, and no more - dont lead!
• Dont help!
• Ask people to explain their actions/statements
  when appropriate
• Dont be afraid to follow the conversation in a
  direction other than the one that was originally
  intended

40
Lets do a user test right here, right now

• Were going to test a Web application MailMan
• 3 examples of people who manage mailing lists and
  would make good test subjects
• Someone who administers one or more mailing lists
  and has used MailMan to do so
• Someone who administers one or more mailing lists
  but has never used MailMan to do so
• Someone who has never administered a mailing list
  or used MailMan
• Today, were looking for the second

41
A Quick Screener

• Everyone, put your hand up
• More formal recruiting screener in packet

• Put your hand down if
• You dont want to participate
• You have participated in a research study in the
  last 6 months
• You write software for a living
• You have never managed a mailing list
• You have used the MailMan mailing list software

42
Come on down!
43
Some Observing Tips

• Imagine theres a mirror in the room
• Pay careful attention to the feedback the
  participants give
• Look for patterns in an individual participants
  actions, and across multiple participants
• Dont get too hung up on one particular phrase,
  comment, or problem that a single user has
• Be a detached observer leave your opinions
  outside the observation room

44
Lets Test!
45
5. Look for trends

• Questions to ask yourself
• Did the users consistently misunderstand
  anything? If so, what?
• Were there any mistakes consistently made? If
  so, what?
• Did they do what you had expected? If not, what
  did they do?
• Did they do things in the order in which you had
  expected? If not, what order did they do them
  in?
• What did they find interesting?
• What did you expect them to find interesting,
  which they did not?

46
5. Look for trends

• More questions to consider
• How many of the tasks were they able to do?
  Which ones did they have the most trouble with?
• When did they look frustrated? What were they
  doing?
• Did the application meet their expectations? If
  not, where did it fail them?
• Were they ever confused? What were they doing?
• Complete list of guidelines for usability testing
  available in your packet

47
Mental Models and Task Analysis
48
Usability testing is fine after the fact, but how
do you start creating a product that works for
your users?

• Learning about peoples needs (conceptual
  research), and then designing your product to
  respond to those needs appropriately
• Our process
• Figure out what users need develop a mental
  model
• Figure out what you have develop a content model
• Match them up
• Use the combined model to guide your products
  feature development

49
What is a Mental Model?

• How an audience thinks about and approaches
• its tasks and goals
• (separate from a computer-mediated experience)

50
What a Mental Model Looks Like

• Collections of tasks in ever-more-general
  groupings

51
What a Mental Model Looks Like

• Consists of Tasks
• The individual tasks that people perform when
  attempting to achieve a larger goal

52
What a Mental Model Looks Like

• Consists of Task Groups
• Tasks for the same goal grouped together

53
What a Mental Model Looks Like

• Consists of Mental Spaces
• The set of goals which together form a complete
  activity

54
How a Mental Model is Used

• Content Slotting

Online Discussion Boards
Proposal Template

• Existing product features are slotted
  underneath to show where your current product
  meets (or doesnt) users needs.

Proposal Submission Form
55
End result Horizon Chart

• Detailed map of your users everyday goals, and
  the individual tasks they undertake to achieve
  them

56
The Process Two Tracks
57
Top track Task Analysis

• Conceptual research that produces a Mental Model
  Diagram
• A deep analysis of user tasks and goals
• Break it down, then build it up

58
Why Perform Task Analysis?

• Stop talking bad design
• To remove the phrase I think from discussions
  about what your users need
• To remove the phrase Someone might want it -- It
  should be an option! from discussions about what
  your users want (well, sometimes)
• And start doing good design
• Helps you figure out what features are important
  to your users, and what they would call those
  features
• Ensures that your product meets those feature
  requirements
• Provides a way to trace back all aspects of the
  interface to the problem your users are trying to
  solve
• Research provides a framework in which good
  design can be done

59
Gathering User Task Data
60
Gather Task Data Prepare for the Interview

• Recruit participants (Just like before)
• Screener
• Friends and family
• Mailing lists
• Select a workflow to explore and questions to
  answer
• What is the goal of your software?
• How do your desired users currently go about
  achieving this goal?
• What are the techniques (on and off the computer)
  they currently use in order to achieve this goal?
• Prepare the discussion guide
• Focus on exploring all the tasks in the workflow
• The key verb is do not feel
• Dont assume the Web or other technological
  solutions

61
Gather Task Data Conduct Interviews

• Use ethnographic inquiry techniques
• Encourage open answers, rather than leading the
  interviewee in any preconceived direction
• Use predefined questions as prompts in a
  conversation, not a verbatim script
• Allow the interviewee to direct the flow of
  conversation
• Interview about 5 people per audience type
• Prepare verbatim transcripts
• 2 options
• Type while talking
• Record and transcribe later
• End Result Detailed notes from a series of
  interviews

62
Next We Analyze the Transcripts
63
Transcript Analysis What Is It?

• An extremely detailed analysis of what your users
  said they do to accomplish their goals
• A depersonalized way to understand your target
  audience
• All users within a particular audience set are
  lumped together
• No individual voice stands out
• Less concerned with sequential order of tasks
  than with sensible grouping of tasks

64
Transcript Analysis How Do You Do It?

• Scan interview transcripts for tasks
• Copy each task to the atomic task table
• Notice patterns across users. Group similar
  atomic tasks together under one task name
• Adjust these groups as the patterns grow and
  shift
• Estimate 4 hours per interview

65
Transcript Analysis Develop Conceptual Groups

• Arrange the tasks into conceptual task groups
  based on
• Steps the users described
• Similarity of tasks
• Do this for each audience, if there are multiple
  audiences
• Compare results between audiences and combine if
  appropriate

66
Yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak yak
yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak yak
yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak...
67
But that seems like a lot of work!

• Well, we did warn you good design takes time
• An alternative approach Two people, an
  afternoon, a large blank wall
• Read notes and make stickies
• One person plucks tasks from the transcript, the
  other writes them down on stickies
• One task per sticky, different colored stickies
  depending on the number of times different people
  mentioned the same task
• Make stickies and move them around until they
  make sense
• Cluster similar stickies on the wall and give
  them a name
• Cluster similar clusters together, and give them
  a name, too
• Voila! Tasks, Task Groups, and Mental Spaces!

68
Transcript Analysis End Result

• A set of conceptual groups and their constituent
  tasks for each audience
• An appreciation for which tasks are common and
  more important

69
Leading To a Diagram of the Users Understanding
70
In other words, Boxes in Visio

• Youve seen this before

71
In other words, Boxes in GNUzio

• Youve seen this before

72
In other words, Boxes in GNUzio

• Consists of Tasks

73
In other words, Boxes in GNUzio

• Consists of Task Groups

74
In other words, Boxes in GNUzio

• Consists of Mental Spaces

75
But what about this part?

• Content Slotting

Online Discussion Boards
Proposal Template
Proposal Submission Form
76
Building a Content Model
Content Model
77
Before You Can Slot Content, You Have to Know
What Content to Slot

• Requires a Content Audit
• A detailed list of all the features,
  sub-features, and preferences for those features
  that exist in your current product
• No product at present? Brainstorm a big list of
  features, determine potential sub-features and
  preferences for those, and go from there
• Content audits arent fun or pretty, but theyre
  extremely valuable
• Software products and Web applications have a
  finite set of features, so the process usually
  only takes a half a day or so
• Operating systems and bloatware take longer
• Be glad youre not auditing Microsoft Word, or a
  corporate Web site with tens of thousands of
  documents scattered everywhere

78
Content Audit - Final Result

• We use Microsoft Excel to track ours
• Be glad youre not auditing Microsoft Excel

79
Build a Content Map (More Boxes in GNUzio)
Gift Basket
Wine
Cheese

• Same type of tool as the mental model diagram
• Turns each feature and preference in the
  spreadsheet into a box
• Lets you understand what you have and want in a
  single glance
• Arrange the chunks in a meaningful way for easy
  reference
• But dont obsess over arranging these
• Well be cutting and pasting them soon enough

CheeseProductDetail
Basket Product Detail
Wine Product Detail
Product Detail
OrganicCheesemaking
Varietal Profile
Specialty Baskest
CategoryInfo
Region Profile
Wine SpectatorReprints
Review
VarietalComparisonChart
Cheese Selector
Tools
80
And now you can do this!

• Content Slotting

Online Discussion Boards
Proposal Template
Proposal Submission Form
81
The Last Step in the ProcessBringing Content
Together with the Mental Model
Align MM Content
82
Comparison of Mental Model to Existing Features

• This is where it begins to come together
• Slot content, functionality, features, and
  preferences, where they support your audiences
  (or audiences) mental model
• Just what it sounds like
• Look at every feature, function, or content area,
  box by box
• Figure out where it matches up to the mental
  model youve built
• Cut and paste it underneath the mental model
• Youll quickly notice that
• Some boxes dont fit anywhere
• Some places in the mental model dont seem to get
  any boxes (though you might have a feature idea
  for them, so add them in!)
• Some boxes fit in more than one place, so
  duplicate them and put them everywhere that seems
  right
• Make sure to address every significant feature
  area

83
Comparison is Very Much a Team Effort

• Group participation is essential in this process
• Need domain expertise to ensure completeness
• Thats what that thing is for, right?
• Was anything left out of the content model?
• Very much in the spirit of Open Source
• But dont let the discussion run too long
• Someone needs to own the process and the final
  decision

84
And now you have your products Horizon Chart

• As you can see, these can get pretty big
• Detailed map of your users everyday goals, and
  the individual tasks they undertake to achieve
  them
• Provides a starting point for discussions about
  user requirements, a way to settle the debate
  about user needs

85
The Horizon Chart and How to Read It

• Ideally, every task in the audiences mental
  model is matched up with content and
  functionality
• Practical That is never the case
• We call the process of moving the practical
  towards the ideal Gap Analysis

86
Gap Type 1 User Needs Not Supported by Features

• Could be an important oversight in the
  development of the product
• Could be be an activity not appropriate for for
  the product

87
Gap Type 2 Content Available But No User Need

• Could be extraneous features not worth
  maintaining (R.O.T.)
• Could be an important way to scratch an itch the
  user didnt know she had

88
So what do we now have?

• A diagram depicting the audiences mental model
  across the top, with supporting features mapped
  below
• Fuzzy user data has developed into a solid,
  rigorous model
• A foundation from which to build the
  applications functionality and feature hierarchy

But how do I turn this into an actual application?
89
How do we turn this

90
Into this?

91
In other words, how do we get from a
well-organized pile of content and features to a
meaningful structured experience?

92
Answer Let the mental model guide the way

• 3 E-Z Steps to your application interface
• 1. Organize information according to user
  expectations
• 2. Organize information based on qualities of the
  features
• 3. Label feature and content areas using familiar
  language

93
Task-based Application Architecture Step 1

• Mental spaces become highest level of navigation

94
Task-based Application Architecture Step 2

• Task groups become the second level

95
Task-based Application Architecture Step 3

• Slotted content and functionality from the
  Comparison is placed in appropriate area

96
Real Life Example iTunes
97
iTunes Step 1

• Mental spaces become highest level of navigation

Play Music
Find Music
98
iTunes Step 2

• Task groups become the second level
• Organize music in playlists
• Look for an Internet radio station

Find Music
99
iTunes Step 3

• Slotted content and functionality from the
  Comparison is placed in appropriate area

• Browse for song by Artist
• Browse for song by Album

100
But that doesnt quite totally work, does it?

• iTunes is not the ideal application because there
  are other issues to take into consideration in
  the interface besides just the users mental
  model
• In this case, users also have assumptions created
  by the design of physical music players, as well
  as by interface innovations developed in
  competing products (ie, playlists)
• All of these considerations come into play when
  designing your applications interface

However, a good task-based interface can be an
innovation in itself
101
Even Better Real Life Example iPhoto
102
iPhoto Step 1

• Mental spaces are the primary interface to the
  application

Share Photos
Get Photos
103
iPhoto Step 2

• Conceptual groups become the second level

Share Photos
104
Caveats

• This is a first-pass at the applications
  feature-level architecture
• All of it will need refinement
• Usability testing, user feedback analysis, and
  log file analysis will all provide valuable
  information to tweak the interface
• Limited in its depth
• Doesnt solve the problem of deep interaction
  development required by some software
• Some tasks dont directly translate to navigation
  nodes
• Some applications dont support a task-based model

105
Things To Remember And Forget

• Remember
• Everything needs to have a place in the design
  but not necessarily only one way to get to it.
• Formality of this process is up to you
• Forget for now
• The underlying code
• The amount of work involved in building or
  changing the features in question
• Any personal attachment to (or dislike for) the
  features in question

106
Feedback Analysis
107
User Feedback is Everywhere

• Support comments, bulletin board posts and
  mailing list chatter are a direct line into the
  thoughts, interests, identities and problems of
  your users
• Often the only kind of interaction Open Source
  developers have with end users apart from
  themselves and other developers
• One problem unsolicited email is biased
• Two kinds of people primarily send feedback email
• Grouches
• Geeks
• Unfortunately, neither ones experience or
  expectations is representative of the bulk of the
  user population

108
However, feedback can be useful

• It helps you see how people think about the
  product as a tool and to learn the language they
  use. This helps you understand their mental
  model of the product and how theyre trying to
  use it
• It reveals a lot about people's expectations
  where those expectations are met and where
  they're not
• It underscores the perceived "points of pain,"
  which helps prioritize issues and determine which
  features to concentrate development efforts on
• It gives you specific questions to ask in your
  research plan and directions for guiding future
  research

109
Steps in Feedback Analysis

1. Collect feedback
2. Read it
3. Code it
4. Analyze it

110
1. Collecting Feedback

• List all of the places where users comment about
  the product
• Support mailing lists
• Message boards
• Email to individual developers
• Gather feedback from as many sources as possible
• Put in a database or a spreadsheet
• 100-200 randomly-selected comments, all from the
  same time period
• Remember where they came from
• Source
• Date

111
2. Read the Feedback

• Gauge the content
• Read from the users perspective (what they know,
  what words they use, etc.)
• Focus on the facts (ignore the emotional content
  of flames and compliments)
• Dont jump to conclusions (a couple of messages
  is not a trend)
• Take comments with a grain of salt

112
2. Read the Feedback (Cont.)

• Consider the source
• Who are they?
• What are their goals?
• How are they approaching the problem?
• The system theyre using (OS, versions, etc.)
• The problem theyre having

113
3. Code it

• Coding a comment is the process of classifying
  it with other similar comments
• Many methods and variations on the idea
• One approach 4 Steps, 2 analysts
• Analyst A works through the messages, creating
  categories until no obviously new categories
  appear. Comments are categorized at the sentence
  level and each category is assigned a name (its
  code). A single sentence can be associated with
  multiple codes.
• Analysts A and B independently categorize the
  same subset of the messages to verify that the
  codes are understandable and complete.
• Analysts A and B compare their categorizations
  and adjust the codes appropriately, creating
  guidelines for their use.
• The rest of the messages (and future messages)
  are divided up amongst as many people as
  necessary and coded.

114
4. Analyze the Support Comments

• Tabulate the categories how many comments appear
  in each category?
• Categories generally indicate
• Problems
• User interest
• User preference
• Regular random sampling and coding can reveal how
  peoples perception of the UE changes with
  updates to the product

115
Log File Analysis
116
Web Log Basics

• All hits to a site are currently logged, or can
  be
• An Apache logfile entry looks like this
• 192.0.34.72 - - 21/Jul/2002045022 -0800 "GET
  /example.html HTTP/1.1" 200 18793

117
Aggregating gives you some idea of popularity
118
Cookies allow you to track order and link
popularity
119
Useful statistics from Web logs

• Number of pages per session
• Duration of session
• First and last pages
• Average path
• Mid-process abandonment
• Features which cause people to invoke help

120
Only Web Sites Keep Logs, right?

• Currently only Web sites and evil spyware tracks
  software feature usage, but it doesnt have to be
  that way
• Nice software can keep usage logs, too
• Open Source is the perfect candidate for learning
  about software usage through event tracking
• Many machines are on the Net full-time
• Open Source and open standards minimize funny
  business
• Open Source interfaces are much more flexible
• There is no big release cycle tied to a marketing
  blitz

121
A proposal for a UE logging system

• Structured event log Distributed survey
  mechanism
• The software maintains a record of which features
  are used and in what order
• The product regularly phones home and uploads
  usage data, while downloading usability,
  preference and satisfaction questions written by
  the developers/designers
• This is an opportunity for UE research innovation
  really only possible under the open source system

122
Potential problems with software UE monitoring

• Security
• Privacy (could easily become a keystroke logger)
• Spoofing (standard attacks)
• Resource use
• Client
• Server

123
Advantages of software UE monitoring

• Straightforward problem
• Limited task
• Many client-server and p2p problems have been
  solved
• With deep hacking possibilities
• Data analysis can be very sophisticated
• Interaction of such a system with other
  technologies (XUL, CORBA, PostgreSQL) presents
  interesting possibilities
• Opportunity to define the field before commercial
  interests
• All Open Source
• Standard XML schema defining interaction
• Standard codebase can be shared among many
  applications
• Full proposal available at http//adaptivepath.com
  /presentations/oscon2002/

124
Thats All, Thanks!

• Contact us anytime
• Mike Kuniavsky ltmikek_at_adaptivepath.comgt
• Lane Becker ltlane_at_adaptivepath.comgt
• Documentation and proposal will soon be online at
  http//adaptivepath.com/presentations/oscon2002/