User Experience Research for Open Software Developers - PowerPoint PPT Presentation

1 / 124
About This Presentation

User Experience Research for Open Software Developers


User Experience Research for Open Software Developers O Reilly Open Source Convention July 23, 8:45AM-12:15PM The Schedule Who we are Introduction Open Source ... – PowerPoint PPT presentation

Number of Views:660
Avg rating:3.0/5.0
Slides: 125
Provided by: MikeKun8


Transcript and Presenter's Notes

Title: User Experience Research for Open Software Developers

User Experience Researchfor Open Software
  • OReilly Open Source Convention
  • July 23, 845AM-1215PM

The Schedule
  • Who we are
  • Introduction
  • Open Source Software and the User Experience
  • User Experience Research Techniques
  • Overview
  • Usability Testing
  • Task Analysis
  • Feedback Analysis
  • Log File Analysis
  • Questions

  • Mike
  • HotWired
  • Put together OpenOffice research plan
  • Wrote a book in StarOffice
  • Wrote a piece for on OSS usability
    in 1999 that got some play ("it's the user,
  • Lane
  • Early DejaNews UI designer
  • Technical editor for OReilly Web and Interface
  • You all
  • What brings you here?

Open Source Software and the User
  • Open Source
  • We all know what Open Software is
  • Some OSS products have great interfacesfor their
    target audience (Apache, perl)
  • This subject of this tutorial
  • Software as a means, not an end
  • Software thats designed to be used by users who
    are not programmers
  • GUI, not CLI
  • All the answers cannot be found in rules and
    guidelines. Each piece of software is different.

OSS and the UE Problem Overview
  1. There arent enough Open Source UI Designers
  2. Usability can be difficult and painful
  3. Pandemic featuritis
  4. Copying without understanding is not enough
  5. Design is tough to do by committee
  6. Running does not equal working
  7. OSS Software is not written for specific audiences

1. Not Enough OSS UI Designers, why?
  • Designers cant code
  • but coders can place interface widgets
  • Designers cant make software by themselves
  • Cultural gap
  • Different academic traditions
  • Design is harder to share than code
  • Seen as window dressing by some

2. Good usability can be painful
  • Many of the little details which improve the
    interface are not exciting or satisfying to work
    on, so they get fixed slowly (if at all).-
    Matthew Thomas, Mozilla UI designer
  • Good UE design requires great attention to detail
  • Solutions are uncertain and often compromises
  • You cant kludge it

3. Pandemic Featuritis
  • EMACS is the granddaddy of OSS bloatware
  • but Mozilla is the reigning monarch
  • Everyone has a favorite feature and wants their
    15 pixels of fame
  • In the name of flexibility
  • every idea becomes a feature
  • every feature has options
  • every option is made a preference
  • The ultimate expression of this is the concept of
    skins, but skins fix nothing


4. Copying doesnt fix anything, either
  • Just because Apple did it doesnt mean its

Copying is not enough
  • Surface issues
  • MS and Apple make plenty of bad UI decisions
  • Example
  • Deeper issues
  • iTunes may have a good interface, but WHY?
  • Good interface design is an expression of good
    architectural design decisions

Copying without understanding is like telephone
  • What is he holding?
  • Fan blade?
  • Cinnamon stick?
  • Sausage?

Copying without understanding amplifies mistakes
Taken from http//
5. Running does not equal working
  • Just because it compiles doesn't mean it works
  • Just because it works doesn't mean it works well
  • Just because it works for you doesn't mean that
    it works for everybody
  • The default matters what you give most users is
    what most users will usually use

6. OS Software is not written for specific
  • You are not your audience!
  • You do not
  • see things like they do
  • know what they know
  • want what they want
  • work how they work
  • This is critical information when designing
    software interaction

So how do you figure out all of these things?
User Experience (UE) Research!
  • The study of what makes peoples lives difficult
    and how to make them easier
  • AbilitiesWhat they can understand and do
  • NeedsWhat people need to make their life easier

Abilities are discovered through Capability
  • Timing After functionality has been implemented
  • Purpose Investigates physical/psychological/memor
    y abilities, reactions
  • Desired Outcome An understanding of which design
    solutions are usable by the desired audience
  • When a problem has been identified and a solution
    created thats designed to satisfy the audiences
    desires, capability research determines whether
    they can actually use it in the manner intended

Needs are discovered through Conceptual Research
  • Timing At the beginning of the design process
  • Purpose Define your audience, understand problem
    scope, investigate peoples needs and current
    methods related to the task your software is
    trying to enable
  • Desired Outcome A foundation for developing a
    product (and therefore the interface to your
    product) based on how your users actually work
  • Conceptual research assumes a good interface
    comes from understanding what your entire user
    base needs
  • It finds ways to create solutions that fit in
    with peoples existing processes and solve
    specific problems theyre having

Research methods should work together
  • Its possible to create a product that works
    well, but solves no problems (The Toilet Cozy)
  • Its possible to create a product that solves
    problems, but people cant use it (The
    Three-Pedal Bicycle)

How is UE Research related to Interface Design?
  • UE research is often lumped in with UI design
  • Its not the same thing
  • UE Research defines problems
  • Interface design, like programming, finds
  • The better the problem definition, the clearer
    the path to the solution
  • (Tangentially interface design is not making
    things pretty, thats graphic design -)

User Research in Open Source Development
  • UE Research and User Experience development in
    Open Source software is in its infancy
  • The OSS model is much more geared toward
    development than design
  • There are UI folks who love to design like you
    like to code, and you should encourage and
    welcome them into the community
  • Open question How can designers participate
    without submitting patches or waging wars in
  • Because there are many more OSS developers than
    designers, coders need to know about this stuff,

UE Research in the Development Process The Ideal
  • Highly iterative
  • Many small steps, rather than a few giant ones
  • Research at every possible step

User Research in the Development Process
  • Developing an iterative approach to user research
    for a product takes time (especially when theres
    no existing place for it in the development
    process, as with most OSS)
  • Bootstrapping an existing development process and
    team is required
  • Linear process
  • Do all research in two lumps conceptual and

Integrating User Research into Development
  • Integrate research into all phases of development
  • No matter what stage a product is in, there's
    some research you can do
  • Make sure research is actionable
  • Time the research for the needs of the product
    and the abilities of the development team
  • Example Don't research button wording before you
    know whether the audience wants the buttons
  • Avoid research paralysis
  • It's OK to make decisions without first asking
    people, just dont make all your decisions that
  • Dont get distracted by research and forget the
  • Be open-minded
  • Users arent stupid, they just dont have as many
    reasons to care as you do

How to get started
  • Dont worry about doing it right
  • UE research scales well
  • Its always worth getting some data
  • Start simple friends and family

The underlying pattern of all UE research
  • The basic process that nearly all UE research
    methods follow
  • Define a test audience
  • Find them
  • Ask them some questions
  • Look for similarities in their responses
  • Generalize similarities into trends
  • Respond to problems through design/development
  • GOTO 2

Usability Testing
Usability Testing An Overview
  • What is usability testing?
  • Way of researching peoples abilities
  • Uses structured interviews
  • Watch people as they use your product
  • When can you use usability testing? Any time!
  • During initial design work
  • Between design iterations
  • To find out why your users complain so much
  • Before a redesign

5-step Process
  1. Define an audience and their goals
  2. Create tasks based on those goals
  3. Bring in some users that match the audience
  4. Have them try the tasks
  5. Look for trends

1. Define an Audience and their Goals
  • You are making a product for some reason.
  • You have decided that some people in the world
    can make their lives better with your idea.
  • These are the questions you need to answer
  • Who are those people?
  • Who will be the largest group of users?
  • What differentiates them from everyone else?
  • Age? Interests? Computer experience? Problems?
  • What does your product do for them?
  • Why are people going to use it?
  • Why is it valuable to them?
  • If you were at a loud party and had 30 seconds to
    describe your product to someone who had never
    heard of it, what would you say?

2. Create Tasks
  • Write down the five most important functions of
    your product
  • Not the features, the functions
  • What problem is being solved?
  • What are the five things that people should
    people be able to do above all others?
  • Basic functions someone should be able to do with
    a word processor
  • Enter text
  • Edit text
  • Format
  • Search and replace
  • Save
  • Create a one to two sentence scenario for each
  • An example of someone using each function, framed
    as a question
  • Written from their perspective

Example Mailing List Management Tasks
  • You have been asked to create a mailing list for
    the California Furby Collector Club. Here is a
    list of the five people in the club. How would
    you go about creating this list?
  • Your Furby list has been a great success and you
    would like to have a record of all of the great
    discussions that happened on the list. You have
    heard that you can set up a permanent collection
    of all of the mail that goes to your mailing
    list. How would you go about setting this up?
  • One of your list members has decided to stop
    collecting Furbys and move on to AIBOs. How
    would you take him off of the list?

3. Find Some Representative Users
  • Narrow down your intended audience ahead of time
  • Pick the people who will give you the best
  • Often a subset of the largest group of users (the
    middle of the bell curve)
  • People whose problems generalize to the largest
    number of other people if you solve the problem
    for them, you solve it for many other groups, too
  • Be specific about who you want
  • Dont recruit people who could have strong
    opinions about the product separate from the
  • Some people know too much
  • Market researchers
  • Designers
  • Software developers

Where to Find Users
  • For simple research friends, family, coworkers
  • Pros cheap, easy
  • Cons bias (they may know too much), not close
    enough to the real target audience
  • For more complex research use a recruiting
  • Pros can get people who know nothing about the
    product, can get people who are exactly your
    audience, can recruit people in a variety of
    geographic locales
  • Cons money

Where To Find Users (Cont.)
  • Some mid-range options
  • Existing user base, customer support inquiries,
    advertise on existing site
  • User groups, email discussion lists
  • Traditional market research means classified
    ads, etc.
  • Start immediately the more time, the better the
    subjects, the better the outcome

Use a Screener
  • A screener is a simple script to help you
    consistently select people to participate in your
  • 20 or so questions that narrow in on the group
    youre after
  • Order questions from generic to specific
  • Be very clear and specific
  • Avoid jargon

Example Profile Mailing List Management Users
  • Demographic Profile
  • Any gender, but prefer roughly even split
  • Any age
  • Technographic Profile
  • Email users
  • Online at least 6 months
  • 8 hours Net use per week, including email and
    Web surfing
  • Don't consider themselves power users
  • Have Internet access at work
  • Behavior
  • Have been on at least one mailing list for at
    least a month
  • Have never managed a mailing list
  • Do not work in marketing, or software development

4. Run the test!
  • Create a comfortable space
  • Set up a typical computer
  • Nothing tricked-out
  • User would expect to do it
  • Ideally, users own computer
  • Prepare the participant
  • Explain what its all about
  • Tell them that theyre evaluating the product and
    that any problems are not their fault
  • Ask them to say all their thoughts aloud
  • Describe the product using your 30-second party

Run the test! (cont)
  • Give them the list of tasks presented as goals
  • Give them exactly as much detail as they need to
    get the job done, and no more - dont lead!
  • Dont help!
  • Ask people to explain their actions/statements
    when appropriate
  • Dont be afraid to follow the conversation in a
    direction other than the one that was originally

Lets do a user test right here, right now
  • Were going to test a Web application MailMan
  • 3 examples of people who manage mailing lists and
    would make good test subjects
  • Someone who administers one or more mailing lists
    and has used MailMan to do so
  • Someone who administers one or more mailing lists
    but has never used MailMan to do so
  • Someone who has never administered a mailing list
    or used MailMan
  • Today, were looking for the second

A Quick Screener
  • Everyone, put your hand up
  • More formal recruiting screener in packet
  • Put your hand down if
  • You dont want to participate
  • You have participated in a research study in the
    last 6 months
  • You write software for a living
  • You have never managed a mailing list
  • You have used the MailMan mailing list software

Come on down!
Some Observing Tips
  • Imagine theres a mirror in the room
  • Pay careful attention to the feedback the
    participants give
  • Look for patterns in an individual participants
    actions, and across multiple participants
  • Dont get too hung up on one particular phrase,
    comment, or problem that a single user has
  • Be a detached observer leave your opinions
    outside the observation room

Lets Test!
5. Look for trends
  • Questions to ask yourself
  • Did the users consistently misunderstand
    anything? If so, what?
  • Were there any mistakes consistently made? If
    so, what?
  • Did they do what you had expected? If not, what
    did they do?
  • Did they do things in the order in which you had
    expected? If not, what order did they do them
  • What did they find interesting?
  • What did you expect them to find interesting,
    which they did not?

5. Look for trends
  • More questions to consider
  • How many of the tasks were they able to do?
    Which ones did they have the most trouble with?
  • When did they look frustrated? What were they
  • Did the application meet their expectations? If
    not, where did it fail them?
  • Were they ever confused? What were they doing?
  • Complete list of guidelines for usability testing
    available in your packet

Mental Models and Task Analysis
Usability testing is fine after the fact, but how
do you start creating a product that works for
your users?
  • Learning about peoples needs (conceptual
    research), and then designing your product to
    respond to those needs appropriately
  • Our process
  • Figure out what users need develop a mental
  • Figure out what you have develop a content model
  • Match them up
  • Use the combined model to guide your products
    feature development

What is a Mental Model?
  • How an audience thinks about and approaches
  • its tasks and goals
  • (separate from a computer-mediated experience)

What a Mental Model Looks Like
  • Collections of tasks in ever-more-general

What a Mental Model Looks Like
  • Consists of Tasks
  • The individual tasks that people perform when
    attempting to achieve a larger goal

What a Mental Model Looks Like
  • Consists of Task Groups
  • Tasks for the same goal grouped together

What a Mental Model Looks Like
  • Consists of Mental Spaces
  • The set of goals which together form a complete

How a Mental Model is Used
  • Content Slotting

Online Discussion Boards
Proposal Template
  • Existing product features are slotted
    underneath to show where your current product
    meets (or doesnt) users needs.

Proposal Submission Form
End result Horizon Chart
  • Detailed map of your users everyday goals, and
    the individual tasks they undertake to achieve

The Process Two Tracks
Top track Task Analysis
  • Conceptual research that produces a Mental Model
  • A deep analysis of user tasks and goals
  • Break it down, then build it up

Why Perform Task Analysis?
  • Stop talking bad design
  • To remove the phrase I think from discussions
    about what your users need
  • To remove the phrase Someone might want it -- It
    should be an option! from discussions about what
    your users want (well, sometimes)
  • And start doing good design
  • Helps you figure out what features are important
    to your users, and what they would call those
  • Ensures that your product meets those feature
  • Provides a way to trace back all aspects of the
    interface to the problem your users are trying to
  • Research provides a framework in which good
    design can be done

Gathering User Task Data
Gather Task Data Prepare for the Interview
  • Recruit participants (Just like before)
  • Screener
  • Friends and family
  • Mailing lists
  • Select a workflow to explore and questions to
  • What is the goal of your software?
  • How do your desired users currently go about
    achieving this goal?
  • What are the techniques (on and off the computer)
    they currently use in order to achieve this goal?
  • Prepare the discussion guide
  • Focus on exploring all the tasks in the workflow
  • The key verb is do not feel
  • Dont assume the Web or other technological

Gather Task Data Conduct Interviews
  • Use ethnographic inquiry techniques
  • Encourage open answers, rather than leading the
    interviewee in any preconceived direction
  • Use predefined questions as prompts in a
    conversation, not a verbatim script
  • Allow the interviewee to direct the flow of
  • Interview about 5 people per audience type
  • Prepare verbatim transcripts
  • 2 options
  • Type while talking
  • Record and transcribe later
  • End Result Detailed notes from a series of

Next We Analyze the Transcripts
Transcript Analysis What Is It?
  • An extremely detailed analysis of what your users
    said they do to accomplish their goals
  • A depersonalized way to understand your target
  • All users within a particular audience set are
    lumped together
  • No individual voice stands out
  • Less concerned with sequential order of tasks
    than with sensible grouping of tasks

Transcript Analysis How Do You Do It?
  • Scan interview transcripts for tasks
  • Copy each task to the atomic task table
  • Notice patterns across users. Group similar
    atomic tasks together under one task name
  • Adjust these groups as the patterns grow and
  • Estimate 4 hours per interview

Transcript Analysis Develop Conceptual Groups
  • Arrange the tasks into conceptual task groups
    based on
  • Steps the users described
  • Similarity of tasks
  • Do this for each audience, if there are multiple
  • Compare results between audiences and combine if

Yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak yak
yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak yak
yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak...
But that seems like a lot of work!
  • Well, we did warn you good design takes time
  • An alternative approach Two people, an
    afternoon, a large blank wall
  • Read notes and make stickies
  • One person plucks tasks from the transcript, the
    other writes them down on stickies
  • One task per sticky, different colored stickies
    depending on the number of times different people
    mentioned the same task
  • Make stickies and move them around until they
    make sense
  • Cluster similar stickies on the wall and give
    them a name
  • Cluster similar clusters together, and give them
    a name, too
  • Voila! Tasks, Task Groups, and Mental Spaces!

Transcript Analysis End Result
  • A set of conceptual groups and their constituent
    tasks for each audience
  • An appreciation for which tasks are common and
    more important

Leading To a Diagram of the Users Understanding
In other words, Boxes in Visio
  • Youve seen this before

In other words, Boxes in GNUzio
  • Youve seen this before

In other words, Boxes in GNUzio
  • Consists of Tasks

In other words, Boxes in GNUzio
  • Consists of Task Groups

In other words, Boxes in GNUzio
  • Consists of Mental Spaces

But what about this part?
  • Content Slotting

Online Discussion Boards
Proposal Template
Proposal Submission Form
Building a Content Model
Content Model
Before You Can Slot Content, You Have to Know
What Content to Slot
  • Requires a Content Audit
  • A detailed list of all the features,
    sub-features, and preferences for those features
    that exist in your current product
  • No product at present? Brainstorm a big list of
    features, determine potential sub-features and
    preferences for those, and go from there
  • Content audits arent fun or pretty, but theyre
    extremely valuable
  • Software products and Web applications have a
    finite set of features, so the process usually
    only takes a half a day or so
  • Operating systems and bloatware take longer
  • Be glad youre not auditing Microsoft Word, or a
    corporate Web site with tens of thousands of
    documents scattered everywhere

Content Audit - Final Result
  • We use Microsoft Excel to track ours
  • Be glad youre not auditing Microsoft Excel

Build a Content Map (More Boxes in GNUzio)
Gift Basket
  • Same type of tool as the mental model diagram
  • Turns each feature and preference in the
    spreadsheet into a box
  • Lets you understand what you have and want in a
    single glance
  • Arrange the chunks in a meaningful way for easy
  • But dont obsess over arranging these
  • Well be cutting and pasting them soon enough

Basket Product Detail
Wine Product Detail
Product Detail
Varietal Profile
Specialty Baskest
Region Profile
Wine SpectatorReprints
Cheese Selector
And now you can do this!
  • Content Slotting

Online Discussion Boards
Proposal Template
Proposal Submission Form
The Last Step in the ProcessBringing Content
Together with the Mental Model
Align MM Content
Comparison of Mental Model to Existing Features
  • This is where it begins to come together
  • Slot content, functionality, features, and
    preferences, where they support your audiences
    (or audiences) mental model
  • Just what it sounds like
  • Look at every feature, function, or content area,
    box by box
  • Figure out where it matches up to the mental
    model youve built
  • Cut and paste it underneath the mental model
  • Youll quickly notice that
  • Some boxes dont fit anywhere
  • Some places in the mental model dont seem to get
    any boxes (though you might have a feature idea
    for them, so add them in!)
  • Some boxes fit in more than one place, so
    duplicate them and put them everywhere that seems
  • Make sure to address every significant feature

Comparison is Very Much a Team Effort
  • Group participation is essential in this process
  • Need domain expertise to ensure completeness
  • Thats what that thing is for, right?
  • Was anything left out of the content model?
  • Very much in the spirit of Open Source
  • But dont let the discussion run too long
  • Someone needs to own the process and the final

And now you have your products Horizon Chart
  • As you can see, these can get pretty big
  • Detailed map of your users everyday goals, and
    the individual tasks they undertake to achieve
  • Provides a starting point for discussions about
    user requirements, a way to settle the debate
    about user needs

The Horizon Chart and How to Read It
  • Ideally, every task in the audiences mental
    model is matched up with content and
  • Practical That is never the case
  • We call the process of moving the practical
    towards the ideal Gap Analysis

Gap Type 1 User Needs Not Supported by Features
  • Could be an important oversight in the
    development of the product
  • Could be be an activity not appropriate for for
    the product

Gap Type 2 Content Available But No User Need
  • Could be extraneous features not worth
    maintaining (R.O.T.)
  • Could be an important way to scratch an itch the
    user didnt know she had

So what do we now have?
  • A diagram depicting the audiences mental model
    across the top, with supporting features mapped
  • Fuzzy user data has developed into a solid,
    rigorous model
  • A foundation from which to build the
    applications functionality and feature hierarchy

But how do I turn this into an actual application?
How do we turn this

Into this?

In other words, how do we get from a
well-organized pile of content and features to a
meaningful structured experience?

Answer Let the mental model guide the way
  • 3 E-Z Steps to your application interface
  • 1. Organize information according to user
  • 2. Organize information based on qualities of the
  • 3. Label feature and content areas using familiar

Task-based Application Architecture Step 1
  • Mental spaces become highest level of navigation

Task-based Application Architecture Step 2
  • Task groups become the second level

Task-based Application Architecture Step 3
  • Slotted content and functionality from the
    Comparison is placed in appropriate area

Real Life Example iTunes
iTunes Step 1
  • Mental spaces become highest level of navigation

Play Music
Find Music
iTunes Step 2
  • Task groups become the second level
  • Organize music in playlists
  • Look for an Internet radio station

Find Music
iTunes Step 3
  • Slotted content and functionality from the
    Comparison is placed in appropriate area
  • Browse for song by Artist
  • Browse for song by Album

But that doesnt quite totally work, does it?
  • iTunes is not the ideal application because there
    are other issues to take into consideration in
    the interface besides just the users mental
  • In this case, users also have assumptions created
    by the design of physical music players, as well
    as by interface innovations developed in
    competing products (ie, playlists)
  • All of these considerations come into play when
    designing your applications interface

However, a good task-based interface can be an
innovation in itself
Even Better Real Life Example iPhoto
iPhoto Step 1
  • Mental spaces are the primary interface to the

Share Photos
Get Photos
iPhoto Step 2
  • Conceptual groups become the second level

Share Photos
  • This is a first-pass at the applications
    feature-level architecture
  • All of it will need refinement
  • Usability testing, user feedback analysis, and
    log file analysis will all provide valuable
    information to tweak the interface
  • Limited in its depth
  • Doesnt solve the problem of deep interaction
    development required by some software
  • Some tasks dont directly translate to navigation
  • Some applications dont support a task-based model

Things To Remember And Forget
  • Remember
  • Everything needs to have a place in the design
    but not necessarily only one way to get to it.
  • Formality of this process is up to you
  • Forget for now
  • The underlying code
  • The amount of work involved in building or
    changing the features in question
  • Any personal attachment to (or dislike for) the
    features in question

Feedback Analysis
User Feedback is Everywhere
  • Support comments, bulletin board posts and
    mailing list chatter are a direct line into the
    thoughts, interests, identities and problems of
    your users
  • Often the only kind of interaction Open Source
    developers have with end users apart from
    themselves and other developers
  • One problem unsolicited email is biased
  • Two kinds of people primarily send feedback email
  • Grouches
  • Geeks
  • Unfortunately, neither ones experience or
    expectations is representative of the bulk of the
    user population

However, feedback can be useful
  • It helps you see how people think about the
    product as a tool and to learn the language they
    use. This helps you understand their mental
    model of the product and how theyre trying to
    use it
  • It reveals a lot about people's expectations
    where those expectations are met and where
    they're not
  • It underscores the perceived "points of pain,"
    which helps prioritize issues and determine which
    features to concentrate development efforts on
  • It gives you specific questions to ask in your
    research plan and directions for guiding future

Steps in Feedback Analysis
  1. Collect feedback
  2. Read it
  3. Code it
  4. Analyze it

1. Collecting Feedback
  • List all of the places where users comment about
    the product
  • Support mailing lists
  • Message boards
  • Email to individual developers
  • Gather feedback from as many sources as possible
  • Put in a database or a spreadsheet
  • 100-200 randomly-selected comments, all from the
    same time period
  • Remember where they came from
  • Source
  • Date

2. Read the Feedback
  • Gauge the content
  • Read from the users perspective (what they know,
    what words they use, etc.)
  • Focus on the facts (ignore the emotional content
    of flames and compliments)
  • Dont jump to conclusions (a couple of messages
    is not a trend)
  • Take comments with a grain of salt

2. Read the Feedback (Cont.)
  • Consider the source
  • Who are they?
  • What are their goals?
  • How are they approaching the problem?
  • The system theyre using (OS, versions, etc.)
  • The problem theyre having

3. Code it
  • Coding a comment is the process of classifying
    it with other similar comments
  • Many methods and variations on the idea
  • One approach 4 Steps, 2 analysts
  • Analyst A works through the messages, creating
    categories until no obviously new categories
    appear. Comments are categorized at the sentence
    level and each category is assigned a name (its
    code). A single sentence can be associated with
    multiple codes.
  • Analysts A and B independently categorize the
    same subset of the messages to verify that the
    codes are understandable and complete.
  • Analysts A and B compare their categorizations
    and adjust the codes appropriately, creating
    guidelines for their use.
  • The rest of the messages (and future messages)
    are divided up amongst as many people as
    necessary and coded.

4. Analyze the Support Comments
  • Tabulate the categories how many comments appear
    in each category?
  • Categories generally indicate
  • Problems
  • User interest
  • User preference
  • Regular random sampling and coding can reveal how
    peoples perception of the UE changes with
    updates to the product

Log File Analysis
Web Log Basics
  • All hits to a site are currently logged, or can
  • An Apache logfile entry looks like this
  • - - 21/Jul/2002045022 -0800 "GET
    /example.html HTTP/1.1" 200 18793

Aggregating gives you some idea of popularity
Cookies allow you to track order and link
Useful statistics from Web logs
  • Number of pages per session
  • Duration of session
  • First and last pages
  • Average path
  • Mid-process abandonment
  • Features which cause people to invoke help

Only Web Sites Keep Logs, right?
  • Currently only Web sites and evil spyware tracks
    software feature usage, but it doesnt have to be
    that way
  • Nice software can keep usage logs, too
  • Open Source is the perfect candidate for learning
    about software usage through event tracking
  • Many machines are on the Net full-time
  • Open Source and open standards minimize funny
  • Open Source interfaces are much more flexible
  • There is no big release cycle tied to a marketing

A proposal for a UE logging system
  • Structured event log Distributed survey
  • The software maintains a record of which features
    are used and in what order
  • The product regularly phones home and uploads
    usage data, while downloading usability,
    preference and satisfaction questions written by
    the developers/designers
  • This is an opportunity for UE research innovation
    really only possible under the open source system

Potential problems with software UE monitoring
  • Security
  • Privacy (could easily become a keystroke logger)
  • Spoofing (standard attacks)
  • Resource use
  • Client
  • Server

Advantages of software UE monitoring
  • Straightforward problem
  • Limited task
  • Many client-server and p2p problems have been
  • With deep hacking possibilities
  • Data analysis can be very sophisticated
  • Interaction of such a system with other
    technologies (XUL, CORBA, PostgreSQL) presents
    interesting possibilities
  • Opportunity to define the field before commercial
  • All Open Source
  • Standard XML schema defining interaction
  • Standard codebase can be shared among many
  • Full proposal available at http//

Thats All, Thanks!
  • Contact us anytime
  • Mike Kuniavsky ltmikek_at_adaptivepath.comgt
  • Lane Becker ltlane_at_adaptivepath.comgt
  • Documentation and proposal will soon be online at
Write a Comment
User Comments (0)