User Experience Research for Open Software Developers - PowerPoint PPT Presentation

Loading...

PPT – User Experience Research for Open Software Developers PowerPoint presentation | free to download - id: 5dc300-M2I3O



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

User Experience Research for Open Software Developers

Description:

User Experience Research for Open Software Developers O Reilly Open Source Convention July 23, 8:45AM-12:15PM The Schedule Who we are Introduction Open Source ... – PowerPoint PPT presentation

Number of Views:503
Avg rating:3.0/5.0
Slides: 125
Provided by: MikeKun8
Learn more at: http://www.adaptivepath.com
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: User Experience Research for Open Software Developers


1
User Experience Researchfor Open Software
Developers
  • OReilly Open Source Convention
  • July 23, 845AM-1215PM

2
The Schedule
  • Who we are
  • Introduction
  • Open Source Software and the User Experience
  • User Experience Research Techniques
  • Overview
  • Usability Testing
  • Task Analysis
  • Feedback Analysis
  • Log File Analysis
  • Questions

3
Introduction
  • Mike
  • HotWired
  • Put together OpenOffice research plan
  • Wrote a book in StarOffice
  • Wrote a piece for sendmail.net on OSS usability
    in 1999 that got some play ("it's the user,
    stupid)
  • Lane
  • Early DejaNews UI designer
  • Technical editor for OReilly Web and Interface
    books
  • You all
  • What brings you here?

4
Open Source Software and the User
ExperienceAssumptions
  • Open Source
  • We all know what Open Software is
  • Some OSS products have great interfacesfor their
    target audience (Apache, perl)
  • This subject of this tutorial
  • Software as a means, not an end
  • Software thats designed to be used by users who
    are not programmers
  • GUI, not CLI
  • All the answers cannot be found in rules and
    guidelines. Each piece of software is different.

5
OSS and the UE Problem Overview
  1. There arent enough Open Source UI Designers
  2. Usability can be difficult and painful
  3. Pandemic featuritis
  4. Copying without understanding is not enough
  5. Design is tough to do by committee
  6. Running does not equal working
  7. OSS Software is not written for specific audiences

6
1. Not Enough OSS UI Designers, why?
  • Designers cant code
  • but coders can place interface widgets
  • Designers cant make software by themselves
  • Cultural gap
  • Different academic traditions
  • Design is harder to share than code
  • Seen as window dressing by some

7
2. Good usability can be painful
  • Many of the little details which improve the
    interface are not exciting or satisfying to work
    on, so they get fixed slowly (if at all).-
    Matthew Thomas, Mozilla UI designer
  • Good UE design requires great attention to detail
  • Solutions are uncertain and often compromises
  • You cant kludge it

8
3. Pandemic Featuritis
  • EMACS is the granddaddy of OSS bloatware
  • but Mozilla is the reigning monarch
  • Everyone has a favorite feature and wants their
    15 pixels of fame
  • In the name of flexibility
  • every idea becomes a feature
  • every feature has options
  • every option is made a preference
  • The ultimate expression of this is the concept of
    skins, but skins fix nothing

9

10
4. Copying doesnt fix anything, either
  • Just because Apple did it doesnt mean its
    right.

iTunes
Rhythmbox
11
Copying is not enough
  • Surface issues
  • MS and Apple make plenty of bad UI decisions
  • Example
  • Deeper issues
  • iTunes may have a good interface, but WHY?
  • Good interface design is an expression of good
    architectural design decisions

12
Copying without understanding is like telephone
  • What is he holding?
  • Fan blade?
  • Cinnamon stick?
  • Sausage?

13
Copying without understanding amplifies mistakes
1567
1750
Today
Taken from http//www.cs.man.ac.uk/playing-cards/t
mfaq/spades.html
14
5. Running does not equal working
  • Just because it compiles doesn't mean it works
  • Just because it works doesn't mean it works well
  • Just because it works for you doesn't mean that
    it works for everybody
  • The default matters what you give most users is
    what most users will usually use

15
6. OS Software is not written for specific
audiences
  • You are not your audience!
  • You do not
  • see things like they do
  • know what they know
  • want what they want
  • work how they work
  • This is critical information when designing
    software interaction

So how do you figure out all of these things?
16
User Experience (UE) Research!
  • The study of what makes peoples lives difficult
    and how to make them easier
  • AbilitiesWhat they can understand and do
  • NeedsWhat people need to make their life easier

17
Abilities are discovered through Capability
Research
  • Timing After functionality has been implemented
  • Purpose Investigates physical/psychological/memor
    y abilities, reactions
  • Desired Outcome An understanding of which design
    solutions are usable by the desired audience
  • When a problem has been identified and a solution
    created thats designed to satisfy the audiences
    desires, capability research determines whether
    they can actually use it in the manner intended

18
Needs are discovered through Conceptual Research
  • Timing At the beginning of the design process
  • Purpose Define your audience, understand problem
    scope, investigate peoples needs and current
    methods related to the task your software is
    trying to enable
  • Desired Outcome A foundation for developing a
    product (and therefore the interface to your
    product) based on how your users actually work
  • Conceptual research assumes a good interface
    comes from understanding what your entire user
    base needs
  • It finds ways to create solutions that fit in
    with peoples existing processes and solve
    specific problems theyre having

19
Research methods should work together
  • Its possible to create a product that works
    well, but solves no problems (The Toilet Cozy)
  • Its possible to create a product that solves
    problems, but people cant use it (The
    Three-Pedal Bicycle)

20
How is UE Research related to Interface Design?
  • UE research is often lumped in with UI design
  • Its not the same thing
  • UE Research defines problems
  • Interface design, like programming, finds
    solutions
  • The better the problem definition, the clearer
    the path to the solution
  • (Tangentially interface design is not making
    things pretty, thats graphic design -)

21
User Research in Open Source Development
  • UE Research and User Experience development in
    Open Source software is in its infancy
  • The OSS model is much more geared toward
    development than design
  • There are UI folks who love to design like you
    like to code, and you should encourage and
    welcome them into the community
  • Open question How can designers participate
    without submitting patches or waging wars in
    Bugzilla?
  • Because there are many more OSS developers than
    designers, coders need to know about this stuff,
    too

22
UE Research in the Development Process The Ideal
  • Highly iterative
  • Many small steps, rather than a few giant ones
  • Research at every possible step

23
User Research in the Development Process
Practical
  • Developing an iterative approach to user research
    for a product takes time (especially when theres
    no existing place for it in the development
    process, as with most OSS)
  • Bootstrapping an existing development process and
    team is required
  • Linear process
  • Do all research in two lumps conceptual and
    ability

24
Integrating User Research into Development
  • Integrate research into all phases of development
  • No matter what stage a product is in, there's
    some research you can do
  • Make sure research is actionable
  • Time the research for the needs of the product
    and the abilities of the development team
  • Example Don't research button wording before you
    know whether the audience wants the buttons
    functionality
  • Avoid research paralysis
  • It's OK to make decisions without first asking
    people, just dont make all your decisions that
    way
  • Dont get distracted by research and forget the
    product
  • Be open-minded
  • Users arent stupid, they just dont have as many
    reasons to care as you do

25
How to get started
  • Dont worry about doing it right
  • UE research scales well
  • Its always worth getting some data
  • Start simple friends and family

26
The underlying pattern of all UE research
  • The basic process that nearly all UE research
    methods follow
  • Define a test audience
  • Find them
  • Ask them some questions
  • Look for similarities in their responses
  • Generalize similarities into trends
  • Respond to problems through design/development
  • GOTO 2

27
Usability Testing
28
Usability Testing An Overview
  • What is usability testing?
  • Way of researching peoples abilities
  • Uses structured interviews
  • Watch people as they use your product
  • When can you use usability testing? Any time!
  • During initial design work
  • Between design iterations
  • To find out why your users complain so much
  • Before a redesign

29
5-step Process
  1. Define an audience and their goals
  2. Create tasks based on those goals
  3. Bring in some users that match the audience
    description
  4. Have them try the tasks
  5. Look for trends

30
1. Define an Audience and their Goals
  • You are making a product for some reason.
  • You have decided that some people in the world
    can make their lives better with your idea.
  • These are the questions you need to answer
  • Who are those people?
  • Who will be the largest group of users?
  • What differentiates them from everyone else?
  • Age? Interests? Computer experience? Problems?
  • What does your product do for them?
  • Why are people going to use it?
  • Why is it valuable to them?
  • If you were at a loud party and had 30 seconds to
    describe your product to someone who had never
    heard of it, what would you say?

31
2. Create Tasks
  • Write down the five most important functions of
    your product
  • Not the features, the functions
  • What problem is being solved?
  • What are the five things that people should
    people be able to do above all others?
  • Basic functions someone should be able to do with
    a word processor
  • Enter text
  • Edit text
  • Format
  • Search and replace
  • Save
  • Create a one to two sentence scenario for each
    function
  • An example of someone using each function, framed
    as a question
  • Written from their perspective

32
Example Mailing List Management Tasks
  • You have been asked to create a mailing list for
    the California Furby Collector Club. Here is a
    list of the five people in the club. How would
    you go about creating this list?
  • Your Furby list has been a great success and you
    would like to have a record of all of the great
    discussions that happened on the list. You have
    heard that you can set up a permanent collection
    of all of the mail that goes to your mailing
    list. How would you go about setting this up?
  • One of your list members has decided to stop
    collecting Furbys and move on to AIBOs. How
    would you take him off of the list?

33
3. Find Some Representative Users
  • Narrow down your intended audience ahead of time
  • Pick the people who will give you the best
    response
  • Often a subset of the largest group of users (the
    middle of the bell curve)
  • People whose problems generalize to the largest
    number of other people if you solve the problem
    for them, you solve it for many other groups, too
  • Be specific about who you want
  • Dont recruit people who could have strong
    opinions about the product separate from the
    interface
  • Some people know too much
  • Market researchers
  • Designers
  • Software developers

34
Where to Find Users
  • For simple research friends, family, coworkers
  • Pros cheap, easy
  • Cons bias (they may know too much), not close
    enough to the real target audience
  • For more complex research use a recruiting
    agency
  • Pros can get people who know nothing about the
    product, can get people who are exactly your
    audience, can recruit people in a variety of
    geographic locales
  • Cons money

35
Where To Find Users (Cont.)
  • Some mid-range options
  • Existing user base, customer support inquiries,
    advertise on existing site
  • User groups, email discussion lists
  • Traditional market research means classified
    ads, etc.
  • Start immediately the more time, the better the
    subjects, the better the outcome

36
Use a Screener
  • A screener is a simple script to help you
    consistently select people to participate in your
    research
  • 20 or so questions that narrow in on the group
    youre after
  • Order questions from generic to specific
  • Be very clear and specific
  • Avoid jargon

37
Example Profile Mailing List Management Users
  • Demographic Profile
  • Any gender, but prefer roughly even split
  • Any age
  • Technographic Profile
  • Email users
  • Online at least 6 months
  • 8 hours Net use per week, including email and
    Web surfing
  • Don't consider themselves power users
  • Have Internet access at work
  • Behavior
  • Have been on at least one mailing list for at
    least a month
  • Have never managed a mailing list
  • Do not work in marketing, or software development

38
4. Run the test!
  • Create a comfortable space
  • Set up a typical computer
  • Nothing tricked-out
  • User would expect to do it
  • Ideally, users own computer
  • Prepare the participant
  • Explain what its all about
  • Tell them that theyre evaluating the product and
    that any problems are not their fault
  • Ask them to say all their thoughts aloud
  • Describe the product using your 30-second party
    speech

39
Run the test! (cont)
  • Give them the list of tasks presented as goals
  • Give them exactly as much detail as they need to
    get the job done, and no more - dont lead!
  • Dont help!
  • Ask people to explain their actions/statements
    when appropriate
  • Dont be afraid to follow the conversation in a
    direction other than the one that was originally
    intended

40
Lets do a user test right here, right now
  • Were going to test a Web application MailMan
  • 3 examples of people who manage mailing lists and
    would make good test subjects
  • Someone who administers one or more mailing lists
    and has used MailMan to do so
  • Someone who administers one or more mailing lists
    but has never used MailMan to do so
  • Someone who has never administered a mailing list
    or used MailMan
  • Today, were looking for the second

41
A Quick Screener
  • Everyone, put your hand up
  • More formal recruiting screener in packet
  • Put your hand down if
  • You dont want to participate
  • You have participated in a research study in the
    last 6 months
  • You write software for a living
  • You have never managed a mailing list
  • You have used the MailMan mailing list software

42
Come on down!
43
Some Observing Tips
  • Imagine theres a mirror in the room
  • Pay careful attention to the feedback the
    participants give
  • Look for patterns in an individual participants
    actions, and across multiple participants
  • Dont get too hung up on one particular phrase,
    comment, or problem that a single user has
  • Be a detached observer leave your opinions
    outside the observation room

44
Lets Test!
45
5. Look for trends
  • Questions to ask yourself
  • Did the users consistently misunderstand
    anything? If so, what?
  • Were there any mistakes consistently made? If
    so, what?
  • Did they do what you had expected? If not, what
    did they do?
  • Did they do things in the order in which you had
    expected? If not, what order did they do them
    in?
  • What did they find interesting?
  • What did you expect them to find interesting,
    which they did not?

46
5. Look for trends
  • More questions to consider
  • How many of the tasks were they able to do?
    Which ones did they have the most trouble with?
  • When did they look frustrated? What were they
    doing?
  • Did the application meet their expectations? If
    not, where did it fail them?
  • Were they ever confused? What were they doing?
  • Complete list of guidelines for usability testing
    available in your packet

47
Mental Models and Task Analysis
48
Usability testing is fine after the fact, but how
do you start creating a product that works for
your users?
  • Learning about peoples needs (conceptual
    research), and then designing your product to
    respond to those needs appropriately
  • Our process
  • Figure out what users need develop a mental
    model
  • Figure out what you have develop a content model
  • Match them up
  • Use the combined model to guide your products
    feature development

49
What is a Mental Model?
  • How an audience thinks about and approaches
  • its tasks and goals
  • (separate from a computer-mediated experience)

50
What a Mental Model Looks Like
  • Collections of tasks in ever-more-general
    groupings

51
What a Mental Model Looks Like
  • Consists of Tasks
  • The individual tasks that people perform when
    attempting to achieve a larger goal

52
What a Mental Model Looks Like
  • Consists of Task Groups
  • Tasks for the same goal grouped together

53
What a Mental Model Looks Like
  • Consists of Mental Spaces
  • The set of goals which together form a complete
    activity

54
How a Mental Model is Used
  • Content Slotting

Online Discussion Boards
Proposal Template
  • Existing product features are slotted
    underneath to show where your current product
    meets (or doesnt) users needs.

Proposal Submission Form
55
End result Horizon Chart
  • Detailed map of your users everyday goals, and
    the individual tasks they undertake to achieve
    them

56
The Process Two Tracks
57
Top track Task Analysis
  • Conceptual research that produces a Mental Model
    Diagram
  • A deep analysis of user tasks and goals
  • Break it down, then build it up

58
Why Perform Task Analysis?
  • Stop talking bad design
  • To remove the phrase I think from discussions
    about what your users need
  • To remove the phrase Someone might want it -- It
    should be an option! from discussions about what
    your users want (well, sometimes)
  • And start doing good design
  • Helps you figure out what features are important
    to your users, and what they would call those
    features
  • Ensures that your product meets those feature
    requirements
  • Provides a way to trace back all aspects of the
    interface to the problem your users are trying to
    solve
  • Research provides a framework in which good
    design can be done

59
Gathering User Task Data
60
Gather Task Data Prepare for the Interview
  • Recruit participants (Just like before)
  • Screener
  • Friends and family
  • Mailing lists
  • Select a workflow to explore and questions to
    answer
  • What is the goal of your software?
  • How do your desired users currently go about
    achieving this goal?
  • What are the techniques (on and off the computer)
    they currently use in order to achieve this goal?
  • Prepare the discussion guide
  • Focus on exploring all the tasks in the workflow
  • The key verb is do not feel
  • Dont assume the Web or other technological
    solutions

61
Gather Task Data Conduct Interviews
  • Use ethnographic inquiry techniques
  • Encourage open answers, rather than leading the
    interviewee in any preconceived direction
  • Use predefined questions as prompts in a
    conversation, not a verbatim script
  • Allow the interviewee to direct the flow of
    conversation
  • Interview about 5 people per audience type
  • Prepare verbatim transcripts
  • 2 options
  • Type while talking
  • Record and transcribe later
  • End Result Detailed notes from a series of
    interviews

62
Next We Analyze the Transcripts
63
Transcript Analysis What Is It?
  • An extremely detailed analysis of what your users
    said they do to accomplish their goals
  • A depersonalized way to understand your target
    audience
  • All users within a particular audience set are
    lumped together
  • No individual voice stands out
  • Less concerned with sequential order of tasks
    than with sensible grouping of tasks

64
Transcript Analysis How Do You Do It?
  • Scan interview transcripts for tasks
  • Copy each task to the atomic task table
  • Notice patterns across users. Group similar
    atomic tasks together under one task name
  • Adjust these groups as the patterns grow and
    shift
  • Estimate 4 hours per interview

65
Transcript Analysis Develop Conceptual Groups
  • Arrange the tasks into conceptual task groups
    based on
  • Steps the users described
  • Similarity of tasks
  • Do this for each audience, if there are multiple
    audiences
  • Compare results between audiences and combine if
    appropriate

66
Yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak yak
yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak yak
yak yak yak yak yak yak yak yak yak yak yakYak
yak yak yak yak yak yak yak yak yak yak yak...
67
But that seems like a lot of work!
  • Well, we did warn you good design takes time
  • An alternative approach Two people, an
    afternoon, a large blank wall
  • Read notes and make stickies
  • One person plucks tasks from the transcript, the
    other writes them down on stickies
  • One task per sticky, different colored stickies
    depending on the number of times different people
    mentioned the same task
  • Make stickies and move them around until they
    make sense
  • Cluster similar stickies on the wall and give
    them a name
  • Cluster similar clusters together, and give them
    a name, too
  • Voila! Tasks, Task Groups, and Mental Spaces!

68
Transcript Analysis End Result
  • A set of conceptual groups and their constituent
    tasks for each audience
  • An appreciation for which tasks are common and
    more important

69
Leading To a Diagram of the Users Understanding
70
In other words, Boxes in Visio
  • Youve seen this before

71
In other words, Boxes in GNUzio
  • Youve seen this before

72
In other words, Boxes in GNUzio
  • Consists of Tasks

73
In other words, Boxes in GNUzio
  • Consists of Task Groups

74
In other words, Boxes in GNUzio
  • Consists of Mental Spaces

75
But what about this part?
  • Content Slotting

Online Discussion Boards
Proposal Template
Proposal Submission Form
76
Building a Content Model
Content Model
77
Before You Can Slot Content, You Have to Know
What Content to Slot
  • Requires a Content Audit
  • A detailed list of all the features,
    sub-features, and preferences for those features
    that exist in your current product
  • No product at present? Brainstorm a big list of
    features, determine potential sub-features and
    preferences for those, and go from there
  • Content audits arent fun or pretty, but theyre
    extremely valuable
  • Software products and Web applications have a
    finite set of features, so the process usually
    only takes a half a day or so
  • Operating systems and bloatware take longer
  • Be glad youre not auditing Microsoft Word, or a
    corporate Web site with tens of thousands of
    documents scattered everywhere

78
Content Audit - Final Result
  • We use Microsoft Excel to track ours
  • Be glad youre not auditing Microsoft Excel

79
Build a Content Map (More Boxes in GNUzio)
Gift Basket
Wine
Cheese
  • Same type of tool as the mental model diagram
  • Turns each feature and preference in the
    spreadsheet into a box
  • Lets you understand what you have and want in a
    single glance
  • Arrange the chunks in a meaningful way for easy
    reference
  • But dont obsess over arranging these
  • Well be cutting and pasting them soon enough

CheeseProductDetail
Basket Product Detail
Wine Product Detail
Product Detail
OrganicCheesemaking
Varietal Profile
Specialty Baskest
CategoryInfo
Region Profile
Wine SpectatorReprints
Review
VarietalComparisonChart
Cheese Selector
Tools
80
And now you can do this!
  • Content Slotting

Online Discussion Boards
Proposal Template
Proposal Submission Form
81
The Last Step in the ProcessBringing Content
Together with the Mental Model
Align MM Content
82
Comparison of Mental Model to Existing Features
  • This is where it begins to come together
  • Slot content, functionality, features, and
    preferences, where they support your audiences
    (or audiences) mental model
  • Just what it sounds like
  • Look at every feature, function, or content area,
    box by box
  • Figure out where it matches up to the mental
    model youve built
  • Cut and paste it underneath the mental model
  • Youll quickly notice that
  • Some boxes dont fit anywhere
  • Some places in the mental model dont seem to get
    any boxes (though you might have a feature idea
    for them, so add them in!)
  • Some boxes fit in more than one place, so
    duplicate them and put them everywhere that seems
    right
  • Make sure to address every significant feature
    area

83
Comparison is Very Much a Team Effort
  • Group participation is essential in this process
  • Need domain expertise to ensure completeness
  • Thats what that thing is for, right?
  • Was anything left out of the content model?
  • Very much in the spirit of Open Source
  • But dont let the discussion run too long
  • Someone needs to own the process and the final
    decision

84
And now you have your products Horizon Chart
  • As you can see, these can get pretty big
  • Detailed map of your users everyday goals, and
    the individual tasks they undertake to achieve
    them
  • Provides a starting point for discussions about
    user requirements, a way to settle the debate
    about user needs

85
The Horizon Chart and How to Read It
  • Ideally, every task in the audiences mental
    model is matched up with content and
    functionality
  • Practical That is never the case
  • We call the process of moving the practical
    towards the ideal Gap Analysis

86
Gap Type 1 User Needs Not Supported by Features
  • Could be an important oversight in the
    development of the product
  • Could be be an activity not appropriate for for
    the product

87
Gap Type 2 Content Available But No User Need
  • Could be extraneous features not worth
    maintaining (R.O.T.)
  • Could be an important way to scratch an itch the
    user didnt know she had

88
So what do we now have?
  • A diagram depicting the audiences mental model
    across the top, with supporting features mapped
    below
  • Fuzzy user data has developed into a solid,
    rigorous model
  • A foundation from which to build the
    applications functionality and feature hierarchy

But how do I turn this into an actual application?
89
How do we turn this

90
Into this?

91
In other words, how do we get from a
well-organized pile of content and features to a
meaningful structured experience?

92
Answer Let the mental model guide the way
  • 3 E-Z Steps to your application interface
  • 1. Organize information according to user
    expectations
  • 2. Organize information based on qualities of the
    features
  • 3. Label feature and content areas using familiar
    language

93
Task-based Application Architecture Step 1
  • Mental spaces become highest level of navigation

94
Task-based Application Architecture Step 2
  • Task groups become the second level

95
Task-based Application Architecture Step 3
  • Slotted content and functionality from the
    Comparison is placed in appropriate area

96
Real Life Example iTunes
97
iTunes Step 1
  • Mental spaces become highest level of navigation

Play Music
Find Music
98
iTunes Step 2
  • Task groups become the second level
  • Organize music in playlists
  • Look for an Internet radio station

Find Music
99
iTunes Step 3
  • Slotted content and functionality from the
    Comparison is placed in appropriate area
  • Browse for song by Artist
  • Browse for song by Album

100
But that doesnt quite totally work, does it?
  • iTunes is not the ideal application because there
    are other issues to take into consideration in
    the interface besides just the users mental
    model
  • In this case, users also have assumptions created
    by the design of physical music players, as well
    as by interface innovations developed in
    competing products (ie, playlists)
  • All of these considerations come into play when
    designing your applications interface

However, a good task-based interface can be an
innovation in itself
101
Even Better Real Life Example iPhoto
102
iPhoto Step 1
  • Mental spaces are the primary interface to the
    application

Share Photos
Get Photos
103
iPhoto Step 2
  • Conceptual groups become the second level

Share Photos
104
Caveats
  • This is a first-pass at the applications
    feature-level architecture
  • All of it will need refinement
  • Usability testing, user feedback analysis, and
    log file analysis will all provide valuable
    information to tweak the interface
  • Limited in its depth
  • Doesnt solve the problem of deep interaction
    development required by some software
  • Some tasks dont directly translate to navigation
    nodes
  • Some applications dont support a task-based model

105
Things To Remember And Forget
  • Remember
  • Everything needs to have a place in the design
    but not necessarily only one way to get to it.
  • Formality of this process is up to you
  • Forget for now
  • The underlying code
  • The amount of work involved in building or
    changing the features in question
  • Any personal attachment to (or dislike for) the
    features in question

106
Feedback Analysis
107
User Feedback is Everywhere
  • Support comments, bulletin board posts and
    mailing list chatter are a direct line into the
    thoughts, interests, identities and problems of
    your users
  • Often the only kind of interaction Open Source
    developers have with end users apart from
    themselves and other developers
  • One problem unsolicited email is biased
  • Two kinds of people primarily send feedback email
  • Grouches
  • Geeks
  • Unfortunately, neither ones experience or
    expectations is representative of the bulk of the
    user population

108
However, feedback can be useful
  • It helps you see how people think about the
    product as a tool and to learn the language they
    use. This helps you understand their mental
    model of the product and how theyre trying to
    use it
  • It reveals a lot about people's expectations
    where those expectations are met and where
    they're not
  • It underscores the perceived "points of pain,"
    which helps prioritize issues and determine which
    features to concentrate development efforts on
  • It gives you specific questions to ask in your
    research plan and directions for guiding future
    research

109
Steps in Feedback Analysis
  1. Collect feedback
  2. Read it
  3. Code it
  4. Analyze it

110
1. Collecting Feedback
  • List all of the places where users comment about
    the product
  • Support mailing lists
  • Message boards
  • Email to individual developers
  • Gather feedback from as many sources as possible
  • Put in a database or a spreadsheet
  • 100-200 randomly-selected comments, all from the
    same time period
  • Remember where they came from
  • Source
  • Date

111
2. Read the Feedback
  • Gauge the content
  • Read from the users perspective (what they know,
    what words they use, etc.)
  • Focus on the facts (ignore the emotional content
    of flames and compliments)
  • Dont jump to conclusions (a couple of messages
    is not a trend)
  • Take comments with a grain of salt

112
2. Read the Feedback (Cont.)
  • Consider the source
  • Who are they?
  • What are their goals?
  • How are they approaching the problem?
  • The system theyre using (OS, versions, etc.)
  • The problem theyre having

113
3. Code it
  • Coding a comment is the process of classifying
    it with other similar comments
  • Many methods and variations on the idea
  • One approach 4 Steps, 2 analysts
  • Analyst A works through the messages, creating
    categories until no obviously new categories
    appear. Comments are categorized at the sentence
    level and each category is assigned a name (its
    code). A single sentence can be associated with
    multiple codes.
  • Analysts A and B independently categorize the
    same subset of the messages to verify that the
    codes are understandable and complete.
  • Analysts A and B compare their categorizations
    and adjust the codes appropriately, creating
    guidelines for their use.
  • The rest of the messages (and future messages)
    are divided up amongst as many people as
    necessary and coded.

114
4. Analyze the Support Comments
  • Tabulate the categories how many comments appear
    in each category?
  • Categories generally indicate
  • Problems
  • User interest
  • User preference
  • Regular random sampling and coding can reveal how
    peoples perception of the UE changes with
    updates to the product

115
Log File Analysis
116
Web Log Basics
  • All hits to a site are currently logged, or can
    be
  • An Apache logfile entry looks like this
  • 192.0.34.72 - - 21/Jul/2002045022 -0800 "GET
    /example.html HTTP/1.1" 200 18793

117
Aggregating gives you some idea of popularity
118
Cookies allow you to track order and link
popularity
119
Useful statistics from Web logs
  • Number of pages per session
  • Duration of session
  • First and last pages
  • Average path
  • Mid-process abandonment
  • Features which cause people to invoke help

120
Only Web Sites Keep Logs, right?
  • Currently only Web sites and evil spyware tracks
    software feature usage, but it doesnt have to be
    that way
  • Nice software can keep usage logs, too
  • Open Source is the perfect candidate for learning
    about software usage through event tracking
  • Many machines are on the Net full-time
  • Open Source and open standards minimize funny
    business
  • Open Source interfaces are much more flexible
  • There is no big release cycle tied to a marketing
    blitz

121
A proposal for a UE logging system
  • Structured event log Distributed survey
    mechanism
  • The software maintains a record of which features
    are used and in what order
  • The product regularly phones home and uploads
    usage data, while downloading usability,
    preference and satisfaction questions written by
    the developers/designers
  • This is an opportunity for UE research innovation
    really only possible under the open source system

122
Potential problems with software UE monitoring
  • Security
  • Privacy (could easily become a keystroke logger)
  • Spoofing (standard attacks)
  • Resource use
  • Client
  • Server

123
Advantages of software UE monitoring
  • Straightforward problem
  • Limited task
  • Many client-server and p2p problems have been
    solved
  • With deep hacking possibilities
  • Data analysis can be very sophisticated
  • Interaction of such a system with other
    technologies (XUL, CORBA, PostgreSQL) presents
    interesting possibilities
  • Opportunity to define the field before commercial
    interests
  • All Open Source
  • Standard XML schema defining interaction
  • Standard codebase can be shared among many
    applications
  • Full proposal available at http//adaptivepath.com
    /presentations/oscon2002/

124
Thats All, Thanks!
  • Contact us anytime
  • Mike Kuniavsky ltmikek_at_adaptivepath.comgt
  • Lane Becker ltlane_at_adaptivepath.comgt
  • Documentation and proposal will soon be online at
    http//adaptivepath.com/presentations/oscon2002/
About PowerShow.com