Defining a Product Vision using Remote Research Methodologies presented to The 14th Annual Usability

1

2

Defining a Product Vision usingRemote Research
Methodologies presented toThe 14th
AnnualUsability Professionals
AssociationConferenceJuly 1st, 2005Mark
SafireDirector of User Experience ResearchSachs
Insights

3
Presentation

• Who am I
• The Products The Vision
• The Company NPD Group
• The Software Products Solution Folders
  PowerView
• The User Profiles
• The Research Challenge
• Remote Moderated Usability
• Definition Benefits Challenges
• Audience Participation Experiences with Remote
  Research
• The Transformation of PowerView through Research
• Research Road Map
• PowerView Research
• Phase 1 Remote Contextual Inquiry
• Phase 2 Remote Moderated Usability
• The Upshot
• QA

4
Who Am I?

• Mark Safire
• Director of User Experience Research, Sachs
  Insights
• Qualitative research provider for NPD Group
  online/offline software products since 2002.
• Spoken at UPA 2003 (Presentation) and UPA 2004
  (Idea Market) on Remote Moderated Usability.
• Also at NYC and Dallas UPA chapters, and for
  Nielsen Norman Group conferences in Las Vegas and
  Amsterdam.
• 2003 Presentation is online at www.sachsinsights.c
  om/services/remotemoderatedusability.html

5
Sachs Insights

• Qualitative research consultancy founded in 1987
  
• Serve automotive, fashion, financial services,
  media, pharmaceutical, technology and travel
  industries.
• Practice Areas Include
• Brand Advertising Research
• New Product Service Development
• User Experience Research
• In our User Experience practice, we conduct
• Concept Process Focus Groups
• Taxonomy Nomenclature Focus Groups
• Usability
• Contextual Inquiry

6
The Company

• A leading provider of sales and marketing
  information based on combined point-of-sale and
  consumer data for over a dozen major industries.
• Apparel, appliances, automotive, beauty,
  cellular, consumer electronics, food and
  beverage, foodservice, footwear, housewares,
  imaging, information technology, movies, music,
  software, toys, travel, video games, ...
• Supplemented by consumer panel data to give a
  more complete picture demographics, etc.
• Data from over 230 retail partners,
  representing over 90,000 doors worldwide
• Clients Manufacturers, Retailers, Analysts
• Quoted in media regularly
• Source www.npd.com

7
The Software Products

• Goals of Software
• Allow customers to analyze data that NPD has
  collected and organized.
• Usage Scenarios
• Monthly reports Same reports, different numbers
• Ad-hoc queries Different reports each time
• Sample Queries for Reports
• What is my market share vs. my competitors?
• How are inkjet printer sales trending over the
  past 3 mo.s?
• What were the top ten products in sold last
  month?

8
The Software Products

• Solution Folders

• PowerView

9
The Software Products

• Solution Folders
• Make Reports from Templates

• PowerView
• Make Reports from Scratch

Less Less Shorter Little (Online)
More More Longer A Lot (Offline)
Flexibility
Room for Error
Learning Curve
Setup Required
10
The Software Products
PowerView
Harder To Learn
Solution Folders
Easier To Learn
More Flexible
Less Flexible
11
The User Profiles Ingoing Assumptions

• Disseminators
• Market Research experts the gatekeepers
• Experienced in performing quantitative analysis
• Pull, analyze, develop reports for others
• Users
• Support personnel, bus. analysts owners, sales
  mktg.
• Less experienced in performing quantitative
  analysis
• Pull, analyze, developer reports for themselves
• Viewers
• Bus. analysts owners, sales mktg.
• Less experienced in performing quantitative
  analysis
• Review reports created by others

12
Research Challenge Finding Respondents

• Some had low incidence
• User profile not yet a significant percentage.
• Geographical dispersion
• Key retailers not clustered
• Expensive and time consuming to fly them in
• 2nd Round of Research Timing
• Week before Christmas

Solution Remote Moderated Usability
13
Usability in a Lab
Respondent
Moderator
Observer(s)
4
14
Remote Moderated Usability
Respondent
Moderator
Observer(s)
7
15
Remote Moderated Usability

• Differences from In-Lab Usability
• Different Locations
• Software Tool
• Teleconference
• Observers usually apart from Moderator

8
16
Remote Moderated Usability

• Benefits
• Recruiting benefits
• Reach geographically dispersed populations
• Reach hard to recruit populations dont leave
  home/office
• Avoid cost and inconvenience of travel for all
  parties involved.
• Get buy-in from stakeholders in distant
  locations

• NPD Research Conditions
• All had access to high-speed Internet connections
• Participants their companies were open to the
  technology
• NPD Group had a WebEx license and in-house
  expertise, used primarily for remote demos and
  help desk support

9
17
Remote Moderated Usability

• Challenges
• Body Language
• Rapport
• Interruptions (phone, visitors, email, IM, )
• Logistics (time zones, ...)
• Security Privacy
• Observer dynamics
• Broadband connection recommended
• Not for newbies less coordinated

10
18
Audience Participation

• Any Experiences? 1-2 Min. Summary
• Your Name, Title, Company
• Research Description Industry, (Client),
  Audience, Objective
• Why Remote Moderated Usability?
• Challenges you faced Tips for dealing with them

19
Remote Moderated Usability Tools Methods

• Many tools available
• NPD done with WebEx Meeting Manager
• Others commercial apps/ASPs to consider
• IBM Lotus Web Conferencing (f.k.a. IBM Lotus Same
  Time)
• Microsoft LiveMeeting (f.k.a. Placeware)
• Citrix GoToMeeting (f.k.a. DesktopStreaming)
• Macromedia Breeze
• Other free or open source solutions to consider
• Microsoft Messenger/MSN Messenger (f.k.a.
  Microsoft NetMeeting)
• VNC (open source various implementations,
  including RealVNC)
• Others kinds of tools
• Full-time Remote Control Apps (See pcmag.com)
• pcAnywhere, GoToMyPC, Timbuktu, etc.

11
20
Research Roadmap

• Solution Folders
• Initial Launch
• In-Lab Interviews
• Optimize Navigation
• Remote Moderated Usability

• PowerView
• Needs Assessment
• Internal Focus Groups
• Remote Contextual Inquiry
• Prototype before Relaunch (version 5.0)
• Remote Moderated Usability

21
PowerView Research

• Phase I
• Remote Contextual Inquiry
• How do they use it?
• They define demo tasks.
• Does it meet actual needs?
• See Role in Work Process
• Ultimate reports delivered
• How easy is it to use?
• First Use / Experienced Use
• Things to change /Things NOT to change

• Phase 2
• Remote Moderated Usability
• Can they use it?
• Use their pre-defined tasks
• Do fixes meet needs?
• Role-play work process
• Observe ease of use.
• First Use / Experienced Use
• Points of pain improved? / Have to relearn a lot?

22
Phase 1, Remote Contextual Inquiry (RCI)

• First time Ad Hoc use
• Even old-timers stumble when trying to create new
  reports.
• Drag drop interface good idea, dont change!
• Inscrutable 8 character field names
• Hard to envision report layout from interface
• Especially for nested reports
• Expert Repeated use
• Gargantuan monthly reports Same thing, different
  numbers
• Each report is run individually, not as a batch
• Typically formatted each time in Excel not
  saved in tool.
• Often requires specific colors for company and
  competitors in for tables and charts.
• Extra columns created during export to show
  suppression.
• Everyone Look Feel dated want standard
  Windows conventions

23
Phase 2, Remote Moderated Usability (RMU)

• First time Ad Hoc use
• Drag drop approach still used.
• Comprehensible, full text, categorized field
  names
• But can change back to 8 char. abbrevs in
  preferences...
• Preview of report layout
• Expert Repeated use
• Run as a batch
• But Favorites menuv confusing metaphor
• Final MyReports
• Save formatting
• No extra columns!
• gray or red italicized numbers with asterisks

24
PowerView, Phase 1, RCI
1) Drag drop a good idea dont change!
2) Inscrutable 8 character field names, without
any categorization
3) Hard to envision report layout from interface
4) Look Feel dated not standard Windows
conventions.
25
PowerView, Phase 1, RCI

• Widget Production Quarterly Trend
• What they need

26
PowerView, Phase 1, RCI
5) Extra columns
6) Formatting not saved
What they get
27
PowerView, Phase 2 RMU
1) Drag drop not changed!
2) Full text, categorized field names
3) Preview of Report Layout
4) Overall Look Feel is Contemporary
28
PowerView, Phase 2 RMU
5) No extra columns
6) Formatting can be saved
29
PowerView Version 5.0

• Validation
• Changes in Prototype were well received.
• Tweaks
• My Favorites was not an intuitive menu name
• Worked for Solution Folders, but not here. Since
  PowerView is not a Web application, the My
  Favorites web metaphor was not effective.
• Solution Changed menu name to MyReports for
  launch.

30
The Upshot
Harder To Learn
PowerView
Solution Folders
Easier To Learn
More Flexible
Less Flexible
31
Contact Information

• Mark Safire
• Director of User Experience Research
• Sachs Insights
• 200 Varick Street, Suite 500
• New York, NY 10014
• www.sachsinsights.com
• msafire _at_sachsinsights.com
• 212-924-1600 x168
• www.remoteusability.com