End-User Research in the Informatics Process - PowerPoint PPT Presentation


PPT – End-User Research in the Informatics Process PowerPoint presentation | free to download - id: 1b7e5b-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

End-User Research in the Informatics Process


Any set of activities where data are collected and analyzed for the purpose of ... of the Consumer') are notoriously unsuccessful at predicting actual behavior. ... – PowerPoint PPT presentation

Number of Views:22
Avg rating:3.0/5.0
Slides: 38
Provided by: tmo72
Learn more at: http://donrickert.typepad.com


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: End-User Research in the Informatics Process

End-User Research in the Informatics Process
  • An Overview and Demonstration n Don Rickert,
    Ph.D. n 4/8/03

What do I mean by Research?
  • Any set of activities where data are collected
    and analyzed for the purpose of making
    generalizations beyond the study (Hughes, 1999)

What Is User Research in the Technology
Development Context?
  • The discipline that is concerned with
  • Translating end user or other consumer insights
    into the development of new systems (or improving
    legacy systems)
  • Who does this research?much interdisciplinary
    cross-fertilization, especially among
  • Psychology
  • Human Factors
  • Information Decision Systems
  • Market/Consumer Research
  • Industrial Design
  • Anthropology

Fundamental Foci of End-User Research in a
Technology Context
  • Figuring out who the target users are
  • Understanding the motivations of these target
  • Uncovering their unmet needs and desires (what is
  • Using the insights gained to support intelligent
    business decisions

Typical Pragmatic Research Questions
  • Will anybody care about this product?
  • Why should they care?
  • Who are these people who will care?
  • What should the product look like?
  • How should it behave?
  • Will people (the ones who care) pay enough for
    the product for us to make money?
  • Bottom line Should we build it at all?

Faith in Reason
  • Historically, the practice of Research involving
    people tends to be characterized by
  • A rational view of human motivation
  • Faith in linguistic measures (the Voice of the

Risks to Gaining Useful Insights From People
  • Asking irrelevant questions
  • Not asking the right questions
  • Asking questions in such a way as to create
    attitudes and opinions where none previously
  • Assuming that what happens in a formal research
    setting applies in the real world
  • Confusing what people are willing to say in a
    group interview with what they really think

More Risks
  • Observing people doing things that they would not
    really do
  • NOT observing them doing the things that they
    really would do
  • Confounding peoples remembered or imagined
    beliefs, attitudes, intentions or experiences
    with actual beliefs, attitudes, intentions and
  • Talking to, surveying or observing the wrong

Reality Check
  • Beliefs, attitudes and intensions expressed by
    people (i.e. Voice of the Consumer) are
    notoriously unsuccessful at predicting actual
  • What people say and what they do are very often

Why Is This So?
  • People are not consciously aware of all of their
    beliefs, attitudes and intensions.
  • People don't really know what they want or need
    or how to articulate it.
  • Motivation is not completely rational.
  • Our thought process is different in different
    emotional states.
  • Peoples memory about an experience is a VERY
    different thing than the experience itself.

The Role of Science
  • Science is
  • A form of inquiry that is capable of revealing to
    us the structure of the world and explaining what
    the world is really like. It seeks to
  • Discover what is so
  • Explain it
  • Science myths
  • It IS possible to measure the all of the key
    variables and thus explain peoples behavior
  • Science is always deductive and involves
    hypothesis testing
  • Good science is always quantitative
  • Statistical significance is the same as
  • Beyond that, there are MAJOR debates on what
    makes for inquiry that can be suitably called

The Common Concerns on Which Most Scientists Agree
  • The notion of Error and the importance of
    avoiding it
  • Reliability
  • Validity

  • Type I Error
  • Failing to observe something that really is true
  • Type II Error
  • Concluding that something is true when it really
  • Type III Error
  • Asking the wrong questions

  • Are your measurements good?
  • E.g. can you repeat your study and consistently
    get the same results?

  • Are you measuring what you think you are
  • Internal Validity
  • Is the effect you observe due to I stimulus that
    you have identified or something else entirely
    (that you dont know about)?
  • External Validity
  • Do the conclusions based on your observations
    apply in the real world?

Good Science boils down to
  • Avoiding error i.e.
  • Failing to observe something that really is true
  • Concluding that something is true when it really
  • Asking the wrong questions
  • By
  • Developing reliable measures
  • Observing the effects of stimuli on outcomes
  • Ruling out threats to validity

All of the methodology stuff that researchers
argue about
  • is really about tactics for controlling error
    furthering reliability and validity, such as
  • Representative samples
  • Qualitative vs. Quantitative
  • Response bias
  • Degrees of freedom
  • Experimental control
  • Situational biases
  • Repeatability

Convergence of Method
  • No research method is perfect.
  • No method, by itself, can ensure reliability and
  • Scientists always employ multiple methods, using
    the strengths of one or more methods to
    compensate for the weaknesses of each.
  • The process is called
  • Convergence of method
  • Or Triangulation

The Most Common Methods Employed by End-User
  • Questionnaires
  • Focus Groups
  • Interviews
  • Usability Testing
  • Real World Observation, such as
  • Ethnography
  • Analysis of Artifacts

  • Key Points
  • Acceptable for getting a pulse on gross
  • I like this.
  • I hate that.
  • Easy way to collect data for quantitative analysis

  • Typical purpose
  • To measure peoples beliefs, attitudes,
    intentions in order to make inferences about
    their behavior
  • To measure peoples experiences to assess their
    satisfaction with experiences of products
  • To obtain factual information about people,
    such as their income or age.
  • Why researchers use them
  • Easy way to collect data for quantitative
  • Easy to get large sample sizes for statistical
  • The risks
  • They measure peoples remembered or imagined
    beliefs, attitudes, intentions or experiences
  • The possibility
  • A questionnaire with a well-documented
    relationship with actual behavior outcomes makes
    powerful quantitative analyses possible.

Focus Groups
  • Typical purpose
  • Same as questionnaires measure beliefs,
    attitudes, intentions and experiences
  • Why researchers use them
  • An assumption that, somehow, group discussion of
    attitudes, beliefs, intentions and experiences
    draws out the truth.
  • Convenience
  • Focus groups are a big business and there are
    lots of facilities.
  • They look rigorous
  • The risks
  • They measure what people are willing to say in a
    group about their remembered or imagined beliefs,
    attitudes, intentions or experiences
  • The possibility
  • The discussion that occurs in focus groups can
    give researchers many insights to better explain
    actual consumer behavior that they observe in
    companion studies.

  • Key Points
  • Can be
  • In a facility or
  • In the field or
  • Over the telephone
  • Do not generate easy to analyze data.

  • Typical purpose
  • You guessed itmeasure beliefs, attitudes,
    intentions and experiences
  • Why researchers use them
  • Probing by a good interview can draw out much
    more information than questionnaires
  • Conveniencecan even be done over the telephone
    or online
  • The risks
  • Same as questionnaires and focus groups, plus
  • Asking questions can create attitudes and
    opinions where none existed before.
  • The possibility
  • Hearing what consumers say can add to insights
    gained from actual observation of behavior.

Usability Testing
  • Key Points
  • What people do in a controlled setting
  • When you are asking the questions
  • Relatively unobstrusive observation of
    semi-realistic behaviors

Usability Testing
  • Typical things that researchers want to measure
    with usability tests
  • Problems with ease of use and other conventional
    quality problems
  • Why researchers use them
  • A notion of scientific rigorcontrolled setting
    with well-defined usage scenarios
  • They can allow relatively unobtrusive observation
    of semi-realistic behaviors
  • They do a good job of uncovering egregious
    problems with conventional quality
  • The risks
  • Type III error
  • the type of error arising from asking the wrong
  • Observing people doing things that they would not
    really do and NOT observing them doing the things
    that they really would do.

The possibility for usability tests
  • When the right research questions are identified,
    usability testing really can help in identifying
    unforseen problems with a product.
  • When combined with other methods such as
    questionnaires, interviews, real-world
    observation, etc. usability testing provides
    compelling data.

  • Key Points
  • On peoples own turf
  • Observation of what they really do
  • Often supplemented by field interviews and
  • Expensive
  • Data analysis is time-consuming.
  • High external validity (generalizable conclusions)

Analysis of Artifacts
  • Usually
  • Customer Service logs
  • Complaint emails
  • Provide good context for understanding actual
    observations of customer behaviors
  • Can lead to wrong conclusions if interpreted
    stand-alone (not in the context of actual
  • I.e. should be done in concert with ethnography
    or other observational research

Ethics in End-User Research
  • The following slides are exerpted from a refereed
    paper entitled The Ethics Problem in Technology
    Companies and What Can Be Done About It
    presented by Don Rickert and Alycen Whiddon at
  • The eighth annual Ethics Technology
    Conference, June 24-25, 2005 at Saint Louis
  • http//

Happy workplaces are a prerequisite for becoming
ethical workplaces.
  • This is all about creating workplaces that are
    conducive to the type of reflection and
    responsibility needed on the part of every
    individual in the making of an ethics-driven
  • See Mihaly Csikszentmihalyis recent book, Good
    Business Leadership, Flow and the Making of
    Meaning (2003)

Good workplaces dont have
  • Periodic staff reductions, especially when
    outsourcing is used to fill the deleted positions
  • Mergers (culture clashes and staff reductions are
    almost always a result)
  • Too few staff combined with aggressive
    development schedules
  • Vague and ever changing goals
  • Frequent reorganizations and reshuffling of staff
  • Dysfunctional politics

Human-Centered Product Design
  • Human-centered design is all about attempting to
    meet the real needs of consumers, not the
    development of new and improved manufactured
  • What is important is that all companies,
    especially technology companies, need an
    organized and empowered group, whose duty it is
    to focus on the actual needs of customers.

The Belmont Report
  • Prescribes the essential responsibilities of
    researchers working with human beings, including
    informed consent, assessment of risks and
    benefits and selection of research participants.
  • Three key ethical principles
  • Respect for Persons
  • Beneficence
  • Justice
  • A copy of the Belmont Report is inclduded with
    this presentation

Focus Responsibility (Making ethical
considerations everyones problem)
  • While the Belmont Report is a tool for focusing
    ethical decision-making for researchers, The
    Social Impact Statement helps engineers and
    others involved in the design of new products.

The Social Impact Statement
  • Attributed to Ben Shneiderman
  • Its explicit purpose is to provide a framework
    for designers to investigate the social impact of
    the systems they are responsible for.
  • Every product development team should prepare a
    detailed Social Impact Statement for whatever it
    is they produce. The report includes the
    following sections
  • Executive summary
  • Description of the system
  • Analysis of the ethical issues (stakeholders,
    principles, risks, etc.)
  • Recommendations for actions, with analysis of the
    possible outcomes
  • Reader's guide to literature on the issues in
    more depth
  • Appendix that describes the methods used to
    collect data and prepare the analysis.

  • What we need is true education in ethical
  • Such education happens in university programs,
    not in two-day workshops.
  • The real hope is with the next generations.
  • Beacons of light
  • Online Ethics Center for Engineering and Science
    at Case Western Reserve (http//onlineethics.org/)
About PowerShow.com