Research Methods - PowerPoint PPT Presentation

1 / 230
About This Presentation
Title:

Research Methods

Description:

Alert you to the different types of methodology available to ... be best carried out by using a main' focus of one element, with a subsidiary focus of another. ... – PowerPoint PPT presentation

Number of Views:14250
Avg rating:5.0/5.0
Slides: 231
Provided by: tonyf
Category:

less

Transcript and Presenter's Notes

Title: Research Methods


1
Research Methods
  • Collecting, Processing and Analyzing Data

2
Aims of the Session
  • The purpose of this session is to
  • Alert you to the different types of methodology
    available to you in your research
  • Make you aware of the different techniques that
    you might use in collecting, presenting and
    analysing data
  • Discuss the different kinds of problems that you
    might encounter in pursuing your research.

3
Contents
  • Developing Your Research Questions
  • The Different Types of Research
  • Selecting Appropriate Research Methods
  • Robustness of Methods
  • Structuring Your Methodology
  • Data Analysis
  • Problems with The Research Process
  • Summary

4
1. Developing your Research Questions
  • This section of the presentation examines what
    you need to do in order to focus your research.

5
The Purpose of Research
  • The purpose of research is to contribute to a
    current academic debate, and possibly to advance
    knowledge in some manner. This means that the
    research that you undertake has to be
  • Embedded in a recognisable field of study, taking
    account of, and drawing on, past research
  • Of interest to other researchers working in the
    same field, and possibly to the wider community
  • Generalisable to more than one individual
    experience or circumstance.

6
Research Questions
  • An important way of ensuring that research is
    well-focussed and purposeful, is to ensure that
    it provides clear, unambiguous, (and hopefully
    informative), answers to particular questions.
  • These are called your Research Questions, and can
    either be answered via the Literature Review (in
    which case the answers will have already been
    discovered by others, but will be novel to you),
    OR via the practical work that you undertake, in
    which case you could well provide novel and
    original answers.
  • This second type or Research Question will drive
    your practical work.

7
Examples of the Different Research Questions
  • Preliminary Questions intended to be answered by
    the Literature Review
  • What are the available technologies for creating
    a database-driven website for use in furniture
    retail?
  • Which currently existing e-commerce websites for
    furniture retailers demonstrate examples of best
    practice?
  • Focussed Questions to be answered by the
    Practical Research
  • Can commercially available web development tools
    be used to create an effective and
    professional-looking e-Commerce website for a
    furniture retailer?
  • What are the most important aspects of a
    furniture retail e-Commerce website from the
    customer point of view?

8
Typical Research Structure
Conduct Literature Review
The process of research is well-documented. This
diagram more or less describes the activities you
need to undertake. What we will do in this
session, is to look at some of the elements, and
how they fit together.
Select Research Questions
Devise Methodology Research Instruments
Apply Methods Instruments
Perform Statistical Analysis
Test Hypotheses Draw Conclusions
9
Getting Started
The first thing that you will do is to make sure
you are well-informed, and to pose pertinent
questions which your practical research will
answer.
Conduct Literature Review
Select Research Questions
Devise Methodology Research Instruments
Apply Methods Instruments
Perform Statistical Analysis
Test Hypotheses Draw Conclusions
10
Defining Your Field of Study (1)
  • You undertake a Literature Review in a particular
    field, in order to ensure that your research is
    embedded within that field, and that you are
    taking account of the methods, issues, results,
    theories and conventions which apply.
  • At the end of the literature review, you will
    have narrowed the field down to a relatively
    small topic within the field, and will have some
    unanswered questions which your practical
    research will try to investigate.
  • These focussed questions form the basis of your
    Practical Research.

11
Defining Your Field of Study (2)
  • Your Research Questions are crucial, as they
    effectively define both the content of your
    practical research and the manner in which you
    carry it out.
  • Your research methodology is designed
    specifically to attempt to answer these questions
  • The data that you collect will be focussed on
    issues relevant to these questions
  • Your analysis of the data will seek to provide
    answers to the questions
  • Your conclusions will summarise the answers.

12
Practical ResearchQuestions
  • Typical research questions might be
  • To what extent is the business community in Wales
    aware of the potential of Bluetooth?
  • Is the software currently available for teaching
    arithmetic to 5 year olds appropriate and
    effective?
  • Are there differences in the way that men and
    women approach the task of writing software?
  • How can historical events be modelled effectively
    using VRML?

13
Defining Your Population
  • When framing your Research questions, you need to
    be clear about what set of objects, people or
    events forms the background population in your
    study.
  • Are you saying, for example that the CAL software
    you produce is designed for all English-Speaking
    people, all men, all Afro-Caribbeans, all
    children under 5, all those who have been
    diagnosed as dyslexic, or simply to
    Afro-Caribbean boys under 5 with specific
    learning difficulties?
  • If you are investigating whether on-line learning
    is effective, is your population students,
    University students, UK University students,
    Liverpool Hope Students, Liverpool Hope Computing
    Students or Liverpool Hope BSc. Computing
    Students?

14
Research QuestionsReferences (SW Library)
  • Lewis, Ian. - So you want to do research! a
    guide for beginners on how to formulate research
    questions. - 2nd ed. - Edinburgh Scottish
    Council for Research in Education, 1997. - (SCRE
    publication 2 .. - 1860030327

15
2. The Different Types of Research
  • Here we look at the different options available
    to us in carrying out the research.

16
Your Research Focus
  • The main focus of your research can be
  • Product-based Research, focusing on producing a
    piece of hardware or software (or the designs for
    them) which is at the cutting edge of a
    discipline, drawing on other researchers ideas,
    best practice and what is feasible. In doing
    this, you may need to explore how the product
    will enmesh with current systems, and existing
    and future technologies.
  • People-based research, focusing on the people who
    interact with the hardware or software, looking
    at issues such as usability, user behaviour,
    compatibility of software with current user
    systems and other HCI issues.
  • Both of these approaches are legitimate, and it
    is possible that in carrying out your research
    you might need to use elements of each one.

17
Product v. PeopleWhich focus?
  • The focus of your research is decided by you.
  • It will depend upon how confident you are in
    creating a product at the cutting edge, or how
    comfortable you will be as a researcher in
    dealing with people.
  • When you frame your research questions, you need
    to ensure that their focus leads you into the
    kind of research that you want to do.
  • There is no right answer, but you may find that
    your research will be best carried out by using a
    main focus of one element, with a subsidiary
    focus of another. For example producing a piece
    of software which securely encodes personal data
    as a self-encrypting and decrypting file stored
    on an ID card, may well need trialling with real
    people.

18
Approaches to Research
  • There are two main approaches to doing research
  • Quantitative Research looks for hard numerical
    data it proceeds by counting, measuring and
    summarising. The goal is to look for statistical
    significance in the results.
  • Qualitative Research takes a soft approach to
    data, and has a more descriptive feel. It
    attempts to get to the heart of the matter,
    exploring individual cases in detail, and seeking
    the reasons for behaviour and explanations for
    events.
  • Both of these approaches are legitimate, and it
    is
  • possible to combine elements.

19
Quantitative v. QualitativeWhich approach?
  • The approach you use will depend upon your topic
    and your research questions.
  • It will also depend upon how comfortable you as a
    researcher feel about using these methods.
  • There is no right answer here, and, as we shall
    see in the rest of this presentation, there may
    be good reasons for adopting a variety of
    methods, which encompass both quantitative and
    qualitative approaches

20
Types of Research
  • There are five main types of research that you
    might consider
  • Experimental Research
  • Survey Research
  • Evaluative Research
  • Observational Research
  • Developmental Research
  • All five of these types can incorporate both
    quantitative and qualitative approaches

21
Experimental Research
  • This is normally quantitative, but can take two
    forms
  • An attempt to produce a piece of hardware,
    software or a combination of both, which is at
    the cutting edge of a discipline.
  • An attempt to investigate and document the
    performance of a particular piece technology in
    specific circumstances.
  • This might involve
  • Creating hardware or software applications
  • Devising detailed tests and evaluation procedures
  • Carrying out rigorous testing
  • Evaluating performance or usability

22
Survey Research
  • This research can be qualitative or
    quantitative in the widest sense, you are
    interviewing people. This might involve
  • An unstructured interview
  • A semi-structured interview
  • An structured interview based on questionnaire
    (face to face, or by telephone)
  • An administered questionnaire
  • A Postal Questionnaire

23
Evaluative Research
  • This is primarily qualitative. Here you are
    trying to assess whether something is of value,
    whether it meets its specifications, or whether
    it is fit-for-purpose. This might involve
  • Developing a list of criteria on which to make
    judgements
  • Finding a suitable set of evaluators
  • Asking the evaluators to examine the object
    against each of the criteria to judge to what
    extent it conforms to expectations, Weighing the
    positives and the negatives, coming to overall
    conclusions
  • Matching these judgments against similar
    judgements made elsewhere in the literature or in
    real life.

24
Observational Research
  • This research normally uses a qualitative
    approach in the widest sense, you are recording
    peoples behaviour. This might involve
  • Participating in a task or situation
  • Making field notes of experiences
  • Asking another person to keep a log or diary of
    their experiences
  • Creating and using an Observation Schedule
  • Making a check-list of occurrences of particular
    events or items.

25
Developmental Research
  • This type of research is primarily quantitative,
    but will almost certainly use qualitative
    approaches broadly you will be developing a new
    application (in its widest sense) for a
    particular situation. This might involve
  • Discussing with or surveying possible users
  • Creation of a novel algorithm, application or
    other piece of software for a particular
    situation or environment
  • Making detailed notes of your experiences as a
    developer.
  • Discussing with other developers their
    experiences and their solutions to particular
    problems.
  • Testing the Algorithm or Software. This might
    involve using a code-checker, or a formal testing
    procedure.
  • Evaluating the software with a group of potential
    users.

26
Research possibilities
27
Other Forms of Research
  • Historical Documentary Research proceeds by
    scrutinising existing materials, both written and
    artefacts, using them as sources of evidence.
  • Action Research is normally conducted in an
    educational or political context. Action is
    taken, monitored, evaluated and then modified for
    the next cycle.
  • Ethnographic Research consists of an in-depth
    study of a cultural phenomenon, in order to
    generate new theory.
  • Case Study Research selects a whole range of
    research methods in scrutinising one particular
    context or situation.

28
Research methods 1References (SW Library)
  • Crabtree, Benjamin F. - Doing qualitative
    research. - London Sage, 1992. - (Research
    Methods for Primary Care 3). - 0803943121
  • Creswell, John W.. - Research design
    qualitative and quantitative approaches / John W.
    Creswell. - Thousand Oaks, Calif London Sage,
    1994. - 0803952554
  • Creswell, John W.. - Qualitative inquiry and
    research design choosing among five traditions
    / J. - Thousand Oaks, Calif. London SAGE,
    1998. - 0761901434

29
Research methods 2References (SW Library)
  • Miller, Delbert Charles. - Handbook of research
    design and social measurement. - 3rd ed. - New
    York David McKay Co . Inc, 1977. - m0859739
  • Research methods in education and the social
    sciences / Research Methods in. - Block 3B
    Research design. - Milton Keynes Open
    University Press, 1983. - (DE304, Block 3B 3B).
    - 0335074235
  • Yin, Robert K.. - Case study research design
    and methods. - Rev. ed. - Newbury Park London
    Sage, 1989. - (Applied social research methods
    series v.5). - 080393470x

30
3. Selecting Appropriate Research Methods
  • The next few slides discuss how you might go
    about selecting your research methods from those
    available

31
Selecting Your Methodology
  • Your research methodology consists of
  • Research Methods (experiment, survey etc.)
  • Research Instruments (questionnaire, tests etc.)
  • Analytical Tools (statistics, inductive or
    deductive methods)
  • When selecting the methodology, you need to be
    aware of
  • The Research Questions you are trying to answer
  • The Population you are trying to generalise to.

32
Factors to Consider
How has previous research in this area been done?
Undertake Literature Review
What methods have been used?
Select Research Questions
What research instruments have been devised?
Devise Methodology Research Instruments
How will the methods instruments be applied?
Apply Methods Instruments
What Statistical tests can be carried out?
Perform Statistical Analysis
What Research Hypotheses can be tested?
Test Hypotheses Draw Conclusions
33
Appropriate methodology
  • Do your Research Questions involve impressions,
    attitudes, opinions, beliefs or knowledge held by
    people?
  • If so, then survey research is appropriate
  • Do your Research Questions involve behaviour,
    actions, reactions to events, circumstances or
    objects?
  • If so, then observational research is appropriate

34
Appropriate methodology
  • Do your Research Questions involve the
    reliability or robustness of hardware, software,
    systems or infrastructure?
  • If so, then an evaluative study is appropriate
  • Do your Research Questions involve the testing of
    hardware or software at the technical level,
    speed, accuracy, security etc?
  • If so, then experimentation is appropriate

35
A Mix of Methods
  • It may be that your research questions overlap
    some of these categories, or different questions
    address more than category.
  • If so, you should consider a mix of methods, that
    ensure that you cover all eventualities. This may
    bring added benefits. (See Triangulation)

36
Mixed MethodsReferences (SW Library)
  • Mixing methods qualitative and quantitative
    research / edited by Julia Bra. - Aldershot
    Avebury, 1995. - 1859721168
  • Tashakkori, Abbas. - Mixed methodology
    combining qualitative and quantitative approaches
    / Abba. - Thousand Oaks, Calif. London Sage,
    1998. - (Applied social research methods series
    v.46). - 0761900705

37
4. Robustness of Methods
  • As well as ensuring that your questions are
    well-focussed, and your methods relevant and
    appropriate, you need to ensure that your methods
    are also Reliable and Valid

38
Reliability Validity
  • Research is Reliable if different methods,
    researchers and sample groups would have produced
    the same results.
  • Research is Valid if the results produced by the
    research are accurate portrayals of the
    situation, explanations are effective, and
    predictions from the research are actually borne
    out by observation.

39
Reliability
  • Research can have poor reliability, if it on one
    or two cases only, or if personal judgement or
    opinion is included.
  • Reliability can be improved if data collection
    methods are made more precise, we have controlled
    experimentation, and we can produce statistical
    summaries.

40
Validity
  • Research can have poor validity if the data
    produced is too far removed the object under
    study, or the respondent. Using detailed, highly
    structured research instruments can lead to
    distortions, by forcing observations into
    categories when they do not fit. Data summaries
    and averaging can also lead to distortions, and
    meaningless generalities. Graphical
    representations and percentages can be highly
    selective and produce biased findings.
  • Validity can be improved by working directly with
    individuals or objects, focusing on specific
    cases, making detailed observations, conducting
    face to face interviews, taking detailed
    measurements in specific circumstances etc.

41
Reliability v. Validity
Valid, but not reliable
42
Reliability v. Validity
Reliable, but not Valid
43
Methodological Trade-Off
  • If you improve validity, you will almost
    certainly reduce reliability.
  • If you improve reliability it will be at the cost
    of reducing your validity.
  • The trade-off is to balance the two so that the
    benefits of using particular methods outweigh the
    losses incurred.

44
Triangulation
  • Triangulation takes its name from the
    navigational method of positioning a ship at sea
    by making two independent observations.
  • The purpose here is to use two distinct
    methodologies, independent of one another to
    confirm that the effects which we are observing
    are real, and not artefacts of the research
    process.

45
Triangulation
  • Triangulation attempts to counter the
    methodological trade-off by using a mix of
    methodologies.
  • If you are using highly structured, statistical
    or measurement-based research, you supplement
    this with detailed observations or face to face
    interviewing.
  • If your research is mainly based on individuals
    or on single items, you ensure that at least part
    of it has some statistical summaries, structured
    observations or questionnaires.

46
Validity ReliabilityReferences (SW Library)
  • Kirk, Jerome. - Reliability validity in
    qualitative research. - Beverly Hills Calif
    Sage Pubns, 1986. - (Qualitative Research Methods
    Series 1). - 0803924704
  • Litwin, Mark S. - How to measure survey
    reliability and validity. - London Sage, 1995.
    - (The Survey Kit 7). - 0803957041

47
5. Structuring Your Methodology
  • This section looks in detail at the techniques
    that you might employ in the Research Activity
    itself.

48
Research Structure
Conduct Literature Review
After framing your Research Questions, and
selecting your methodology, you should test that
this is going to work by conducting a small-scale
Pilot Study
Select Research Questions
Devise Methodology Research Instruments
Apply Methods Instruments
Perform Statistical Analysis
Test Hypotheses Draw Conclusions
49
5a Pilot Studies
  • A pilot study is a set of preliminary
    investigations procedures carried out prior to
    the main research, to ensure that the research is
    possible and can proceed without hitches.

50
Pilot Study (1)
  • In almost every type of Research, you will need
    to devise or adapt some sort of Research Method
    or Instrument.
  • This might be a measuring procedure, a set of
    evaluation criteria, an interview procedure or
    questionnaire, or an observational method or
    schedule.

51
Pilot Study (2)
  • Your procedure or instrument should be based on
    best practice from previous research.
  • It is unlikely that you will find exactly what
    you need you will be forced to adapt or amend
    it.
  • This means that you will need to conduct a Pilot
    Study to test whether the new instrument is
    fit-for-purpose.

52
Pilot Study (3)
  • The main purpose of a Pilot Study is to iron out
    bugs in procedures, or to check that
    instruments work.
  • The size of the pilot study will depend on how
    inventive you needed to be.
  • If your procedures are almost entirely of your
    own devising, then you will need a fairly
    extensive pilot study to check them.
  • If you have lifted methods from the literature,
    than your pilot can be quite small.

53
Pilot Study (3)
  • With questionnaires observation schedules, you
    will need to check that individual items are
    giving you expected results.
  • With test procedures, you need to check that you
    can actually do what you have said that you are
    going to do.
  • With evaluation criteria, you need to use these
    in a limited context, to see that they are
    workable and effective.

54
Pilot Study (4)
  • As a result of the Pilot Study, you need to
    evaluate procedures and instruments, making
    amendments where necessary.
  • The Pilot Study stage will be part of your
    research you will need to write this up,
    reporting on how your methods were adapted and
    improved as a result.

55
Carrying Out the Research
Undertake Literature Review
Select Research Questions
Devise Methodology Research Instruments
Here we will examine how particular methods and
instruments can be applied in the research
situation.
Apply Methods Instruments
Perform Statistical Analysis
Test Hypotheses Draw Conclusions
56
5b A detailed look at some Research methods
  • Experimental Design
  • Evaluative Research
  • Observational Research
  • Survey Research
  • Developmental Research

57
5c.1 Experimental Design
  • This section looks at the different ways in
    which you can conduct experiments. Note that you
    do not need to be doing pure experimental
    research to adopt these methods.

58
Experimental Designs
  • You may need to think about experimental design
    even if you are doing types of research other
    than Experiments
  • These designs occur where you have made some
    change, and are trying to find out its effect.
  • The net result is that you are comparing one
    thing, or one group with another thing or group.

59
Experimental Designs
  • The terminology for experimental designs comes
    from agricultural experiments.
  • We have different treatments which we apply to
    different groups
  • We control the groups for different factors

60
Experimental Designs
  • Experiments normally involve independent and
    dependent variables
  • Independent variables are factors that can be
    controlled for, like temperature, file sizes,
    age, gender etc.
  • Dependent variables are those factors which will
    change as a result of altering the independent
    variables.
  • For example, download times will increase as a
    result of increasing file sizes.
  • Download time dependent variable,
  • File size independent variable

61
Pre-Test/Post TestControl Group Design
  • This is the classic experimental design.
  • It allows you to split your sample into two
    distinct parts (A B)
  • You give the same test to the two groups before
    you start
  • You treat one of the groups
  • You apply the same test afterwards.

Group A (treated)
Group B (untreated)
Pre-test
Pre-test
Treatment
Post-test
Post-test
62
Control Group
  • The Control Group (B), is one which is the same
    in all respects, except for the fact that we make
    no changes.
  • We can use the test measurements on the control
    group as a benchmark for any changes we make to
    the treatment group (A)

63
Example 1 Server Testing
  • Suppose we wish to check whether a firewall is
    effective in blocking external attacks.
  • Set up System A System B on two different
    servers, running near-equivalent internal
    external programs.
  • Devise a test which includes a full range of
    possible attacks apply to both A B. Take
    series of measurements observations.
  • Incorporate firewall into System A
  • Reapply test to both systems, which are again
    running near-equivalent programs.

64
Example 2 Software Design
  • Suppose we wish to find out whether
    anthropomorphic agents are useful in
    communicating with novice users of a website.
  • Set up Website A Website B with
    near-equivalent structure, but different
    content.
  • Devise a procedure which asks two equivalent sets
    of users to make explorations of the websites
    apply to both A B. Make a series of
    observations
  • Incorporate Agents into Website A
  • Re-apply procedure, asking two more equivalent
    sets of users to make the same explorations of
    both Websites. Again make observations.

65
Example 3 On-Line Learning
  • Suppose we wish to find out whether training is
    effective in helping students cope with the
    demands of on-line learning.
  • Set up group A group B with near-equivalent
    members.
  • The two groups to undertake a short programme of
    learning on-line, which incorporates a short
    evaluation and a knowledge test.
  • Give Training to Group A
  • The two groups to undertake a further short
    programme of learning on-line, again
    incorporating a short evaluation and knowledge
    test.

66
Factorial Designs
  • This is where we examine the effects of two or
    more independent factors simultaneously.
  • For two factors, we would need 4 groups
  • Group A (Control no treatment)
  • Group B (Factor 1 treatment only)
  • Group C (Factor 2 Treatment only)
  • Group D (Factors 1 2 Treatments)
  • Clearly, this is going to increase the complexity
    and size of the research, but it has the added
    benefit of producing verifiable results in cases
    where two factors interact to produce interesting
    effects.

67
Other Design Variants
  • Post-Test only Control Group Design
  • Here we assume that both groups are the same (but
    do not test that assumption). We simply apply the
    treatment to one group, and apply the test to
    both groups.
  • Matched Pairs Design
  • The individuals we use in the test groups
    (subjects) are matched for characteristics which
    are likely to affect the outcome (gender, age,
    level of education, ethnicity etc.)

68
Experimental DesignsReliability Issues
  • For high reliability, we need to build in strict
    controls over each of the independent variables
    affecting the outcomes of the experiment.
  • We will also need to ensure that any measurements
    taken of the dependent variables are as accurate
    as possible.
  • We also need to ensure that our methodology is
    clear and replicable.

69
Experimental DesignsValidity Issues
  • For high validity, we need to conduct the
    experiment in as natural a setting as possible,
    and in as near as exact a match to the
    circumstances in which the events or objects
    would normally operate.
  • If a sample is being used, then the sample
    (whether time, events, people or objects) should
    be as representative of the background
    population as possible, and we should make
    detailed observations of how the events unfold as
    well as the final measurements.

70
Experimental DesignReferences (SW Library)
  • Campbell, Donald T., Donald Thomas, 1916-. -
    Experimental and quasi-experimental designs for
    research / Donald T. Campbell. - Boston London
    Houghton Mifflin, 1963. 0395307872
  • Field, Andy. - How to design and report
    experiments / Andy Field, Graham Hole. - London
    SAGE, 2003. - 0761973826
  • Miller, Steve. - Experimental design and
    statistics. - London Methuen, 1975. -
    (Essential Psychology A8). - m0805407

71
5c.2 Evaluative Research
  • This section looks at the methods and issues
    surrounding evaluation. You may need to use such
    techniques if evaluation is implicit in your
    research questions

72
Evaluative Research
  • Evaluative Research covers those cases where you
    are attempting to compare whether one procedure
    or object is better or more effective than
    another procedure or object, or to determine
    whether a particular procedure or object is
    fit-for-purpose.
  • Evaluation normally needs to be done against a
    set of criteria, which have been established as
    valid and reliable in this context. You would
    normally produce an evaluation form to be
    completed by respondents.

73
User Evaluation
  • Most evaluative research will attempt to involve
    the potential users of the software at some
    point.
  • This will provide very useful information on how
    easy the software is to use for different types
    of user, what issues arise in day-to-day use of
    the software, and whether or not the users feel
    that the software is intuitive, user-friendly and
    well-designed.
  • It is possible to use a qualitative form of user
    evaluation, where a user is asked to use a piece
    of software, and then interviewed about
    experiences this should be used with care, as
    while the information this provides is very
    useful, it can only be indicative.

74
Using Expert Judges
  • Using Expert Judges is often an extremely useful
    method in evaluating a piece of software.
  • Individual users may not have the experience,
    insight or knowledge to provide detailed
    information on every aspect of a piece of
    software.
  • One of the main issues in using this method, is
    to establish the credibility of the experts, and
    the relevance of their experience to the software
    to be evaluated.
  • Expert Judges may also be used qualitatively, by
    asking them to evaluate a piece of software
    according to whatever criteria they feel to be
    appropriate this in general is far more valid
    than asking individual users to comment.

75
Heuristic Evaluation
  • Heuristic evaluation is a usability engineering
    method for finding the usability problems in a
    user interface design so that they can be
    attended to as part of an iterative design
    process.
  • The method involves a small set of evaluators
    examining the interface and judging its
    compliance with recognized usability principles
    (the "heuristics").
  • The evaluation is performed initially by
    individual evaluators working alone. Only after
    all evaluations have been completed are the
    evaluators allowed to communicate and have their
    findings aggregated.
  • This procedure ensures independent and unbiased
    evaluations from each evaluator.
  • http//www.useit.com/papers/heuristic/heuristic_e
    valuation.html

76
Evaluation Criteria
  • To establish criteria for evaluation, we need to
    break the topic down into individual elements,
    and state the different sub-topics on which the
    item is to be evaluated.
  • Alongside this, we will normally state specific
    questions which will need to be answered in order
    to judge the item against the criterion.
  • The answers to the questions may involve
    measurements, counts, assessments on subjective
    scales, statements of fact or possibly even
    opinion.

77
Example Website Evaluation Criteria
  • Design
  • Is the use of colour acceptable?
  • Are the elements in harmony?
  • Navigation
  • Do the links work?
  • How many links per page?
  • Content
  • How many words graphics on page?
  • Is the text interesting informative?
  • Interactivity
  • What interactive features are used?
  • Do these improve communication with the user?
  • Coding
  • What scripting is used? Is it clearly annotated?

78
Example Website Evaluation Form
  • 1. Design
  • Colour Safe Pallet used? Yes/ No
  • Harmony of elements on page good/ neutral/ poor
  • 2. Navigation
  • Number of links on the page ____
  • Number of working links ____
  • 3. Content
  • Total size of graphics files on page ____ MB
  • Is the text boring/ dull/ neutral/
    interesting/ fascinating
  • 4. Interactivity
  • Tick and/or name all interactive features used
  • rollovers dynamic images image maps ______,
    ______
  • 5. Coding
  • What DTD has been used? _______________________

79
Evaluation CriteriaValidity Issues
  • For criteria to be valid, each of the criteria
    elements need to have face or content validity
    the questions posed by the criterion needs to be
    relevant to the object, and relates to a feature
    of it
  • We also need to be able to answer the questions
    in an objective manner, without recourse to
    guessing, or giving an impressionistic response.

80
Evaluation CriteriaReliability Issues
  • For criteria to be reliable, each of the criteria
    elements need to be clear and unambiguous, so
    that different assessors would interpret the
    criteria in the same way.
  • We also need to ensure that repetition of the
    evaluation exercise will yield results which are
    not dissimilar to one another.

81
Evaluation MethodsReferences (SW Library)
  • Britain, Sandy. - A framework for pedagogical
    evaluation of virtual learning environments. -
    Manchester Joint Information Systems Committee,
    1999. - (JISC Technology Applications Programme
    r.. - M0000712EL
  • Broadbent, George Ernest. - The role of
    evaluation in user interface design a practical
    implementation. - Liverpool University of
    Liverpool, 1997. - p7270369
  • Redmond-Pyle, David. - Graphical user interface
    design and evaluation (GUIDE) a practical
    process. - London Prentice Hall, 1995. -
    013315193x
  • Smeltzer, Nicholas. - Critical analysis of the
    design and evaluation of a computer-based
    project. - Liverpool University of Liverpool,
    2001. - M0002704LO

82
5c.3 Observational Research
  • This section describes what you need to do if
    your research involves making detailed
    observations of people, events or objects.

83
Observational Research
  • In this case you are trying to determine
  • Either How an individual or group of individuals
    react, interact or behave in particular
    circumstances
  • Or How software or hardware performs when used
    by particular groups of people

84
Participant Observation
  • In this case, the researcher becomes one of the
    subjects, and works alongside the subject,
    monitoring behaviour and interacting with them.
  • ProsResearcher can fully understand what the
    issues are. Can get real validity to research.
  • Cons Researcher can influence the research,
    alter opinions. Highly subjective unreliable.
    Taking notes is difficult relies on good memory.

85
Non-Participant Observation
  • In this case, the researcher studies the
    situation apart from subjects, taking notes,
    monitoring behaviour and observing interactions.
    The use of audio video recording is useful
    here.
  • ProsResearcher can get overview of the
    situation, and achieve objectivity
  • Cons Researcher can only see resulting
    behaviours, not what is causing it or why it is
    happening.

86
Observation Schedules
  • An observation schedule can be a simple list of
    things to look for in a particular situation
  • It can be far more complex a minute by minute
    count of events such a mouse-clicks or verbal
    interactions between subjects.

87
Observation ScheduleAn Example
  • Observation of subject using Information Portal
  • Subject M / F Age 18-21 / 21-30 / 31-50 / 50
  • Date _________ Time _________
  • Selected Navigation Tool Mouse /Keyboard/
    TouchScreen
  • First 5 pages visited in order __ __ __
    __ __
  • Time to obtain to obtain required information
    ___ min ___ sec
  • Total Number of Pages visited __
  • Feedback from subject
  • very positive / positive / neutral / negative /
    very negative

88
Observation SchedulesSome Reliability Issues
  • For an observation schedule to be reliable, it
    should require structured documentation of
    events.
  • This will involve such things as checklists,
    minute-by-minute categorisation of activity and
    numerical data such as frequencies of occurrence
    and time intervals. Timings should have a clear
    start and end points.
  • The schedule should leave little room for
    subjective judgement.

89
Observation SchedulesSome Validity Issues
  • For an observation schedule to be valid, it
    should refer to events which actually happen
    each of the events on the sheet should be
    possible, and likely to occur.
  • Timings should be possible to take, and not
    interfere with other observations which should be
    made.
  • There should be room for observations which
    enrich the data by adding detail to the
    numbers, offering explanation and illumination.

90
Observational ResearchReferences (SW Library)
  • Harding, Jacqueline. - How to make observations
    assessments / Jackie Harding and Liz
    Meldon-Smith. - 2nd ed. - London Hodder
    Stoughton, 2000. - 034078038x
  • Robertson, Kevin. - Observation, analysis and
    video / editors Anne Simpkin and Penny
    Crisfield. - Leeds National Coaching
    Foundation, 1999. - 1902523164
  • Simpson, Mary. - Using observations in
    small-scale research a beginner's guide / Mary
    Simpson. - Glasgow Scottish Council for
    Research in Education, 1995. - (SCREpublications
    16 130). - 1860030122

91
5c.4 Survey-Type Research
  • This section describes what you need to do in
    order to use human respondents to provide you
    with information.

92
Survey methods
  • There are two distinct elements here
  • The interview techniques that you adopt in order
    to elicit information from people
  • The sampling methods that you adopt in order to
    select respondents for interview
  • You will need to make rational choices for both
    of these elements, depending upon the focus of
    your research and the population under study.

93
Interviewing
  • We interview people face-to-face in order to try
    to find out exactly what they think.
  • With the right questions, people respond with
    high quality information.
  • The data that you get can be of high validity,
    since you have access to respondents own words.

94
Types of Interviews
  • Unstructured Interview
  • Structured Interview
  • Semi-Structured Interview
  • Administered Questionnaire

95
Open or Closed?
  • When interviewing respondents, the main choice
    facing the researcher is whether to use open
    questions, which leave the respondent free to
    answer in any way they think fit, or closed
    questions which force the respondent to make
    particular choices, pre-determined by the
    researcher.
  • OPEN What is your experience of chat rooms?
  • CLOSED Do you think chat rooms should be
    monitored? (Yes/No/Maybe)

96
Open or Closed?Validity Reliability Issues
  • In general, the data from open questions is
    richer, more illuminating, and more valid, as it
    can illustrate clearly why particular subjects
    think the way that they do, or why they behave in
    particular ways.
  • Data from closed questions however, is more
    amenable to statistics. It is easier to
    summarise, spot trend and make comparisons. In
    general the data is more reliable.
  • A good strategy is to use a mixture of both open
    and closed questions

97
Unstructured Interviews
  • Unstructured Interview
  • Interviewer has no set agenda the object is to
    get the respondent to talk freely about various
    topics.
  • Pros Can be high quality data, often get real
    insights into what people really think.
  • Cons Can be difficult for the novice researcher
    to carry out need to be good at steering the
    conversation without forcing it. Time consuming
    can only carry out a small number. Difficult to
    analyse

98
Semi- Structured Interview
  • Interviewer has a formal list of topics, which
    sets the agenda however, the interviewer is free
    to take these in different orders, or to return
    to topics at different points.
  • Here we include the idea of focus groups, where
    a facilitator encourages the discussion of a
    particular topic.
  • Pros Data provided can be almost as good as
    unstructured interview, but in a more focussed
    manner.
  • Cons Can miss important ideas, because agenda
    set beforehand interviews can also be time
    consuming difficult to manage analyse
    afterwards.

99
Structured Interviews
  • Interviewer has a list of topics to be taken in a
    particular order questions are written, and read
    out.
  • Pros You can make good comparisons between
    different respondents you also will have access
    to respondents thinking.
  • Cons Unless you have done some preliminary
    investigations, and extensively piloted the
    interviews, you may miss lots of important data.
    Takes time to do properly can only carry out a
    few.

100
Administered Questionnaire
  • Interviewer has devised a list of questions which
    are read out some multiple choice, some
    open-ended. Filled in either by respondent or
    interviewer.
  • Pros Interview can be quite brief easy to make
    comparisons, data amenable to statistical
    analysis.
  • Cons Data may be warped by the choice of
    questions. Respondent may be forced into giving
    fabricated answers.

101
Questionnaires
  • There are many different types of questionnaire,
    depending upon what you are trying to find out.
  • As well as gathering factual data about the
    person, you may be trying to explore their
  • Knowledge
  • Beliefs
  • Attitudes
  • Opinion
  • Behaviour

If you need to design a questionnaire, see for
example http//www.leeds.ac.uk/iss/documentation/
top/top2/index.html
102
Question Types
  • Factual Data
  • Binary (Yes/No)
  • Single Selection from Categories
  • Multiple Selection from Categories
  • Attitude Scale items
  • Focussed items to elicit categories
  • Conditional Questions for routing
  • Open-ended Questions
  • Self-Reporting

103
Writing Questions
See http//www.analytictech.com/mb313/principl.ht
m
  • Put Factual Data (demographics) at end
  • Put Explanations and Disclaimers at the start
  • Keep questions as brief as possible
  • Use non-technical language where possible
  • Avoid leading respondents towards particular
    answers
  • Use direct questions, no hypotheticals
  • Use simple questions, no portmanteaus
  • Use a mixture of positively and negatively worded
    questions
  • Verify data by asking same question in different
    ways.

104
Questionnaire Example
  • 1. Which operating system are you currently
    using
  • Linux Windows Other
  • 2. How would you rate the operating system?
  • very poor poor good very good
  • 3. Circle the tasks which you use your computer
    to do
  • Word processing Internet Access Program
    Development
  • 4. Estimate the number of hours you use the
    computer for each week
  • ______ hours
  • 5. Do you use broadband?
  • Yes No

105
Protocol
  • Whatever your interviewing method, as part of
    the ethical constraints on Hope Researchers, you
    are required to do the following
  • Explain to the respondents what the research is
    about.
  • Tell them that they will not be identified by
    name in the final report, and that any views that
    they express will be in confidence.
  • Explain to them that if there are questions with
    which they feel uncomfortable, they do not have
    to answer.

106
Attitude Measurement
  • You may wish to incorporate attitude
    measurement into your questionnaire as either a
    major or a minor feature.
  • There are several different types of scales which
    can be used to elicit numerical measurements of
    attitude
  • Likert Scaling
  • Thurstone Scaling
  • Guttman Scaling
  • Semantic Differential Scaling
  • See http//www.socialresearchmethods.net/kb/scalg
    en.htm

107
Likert Scales (1932)
  • This consists of items like I would not trust a
    banks website to keep my details secure
  • Respondents are asked to use a scale such as
    1Strongly disagree 2disagree 3neutral
    4agree 5strongly agree
  • Scales can vary 0-4, 1-7 etc.
  • Some proponents suggest removing the neutral
    category to force a choice, but this can reduce
    validity.
  • Scores on individual items are totalled to obtain
    the respondents score this is often shown as an
    average.

108
Thurstone Scales (1928)
  • This pre-ranks 11 statements with numerical
    values 1-11 each statement carries a numerical
    score, for example
  • 1 Teleworking is an impossible concept
  • 2 Teleworking may be OK in exceptional cases
  • 8 If offered the opportunity, I would try
    teleworking
  • 11 In 20 years time, everyone will be
    teleworking
  • Clearly, lots of preparatory work needs to be
    done to generate and rank the statements.
  • A respondents score, is the average numerical
    value of all the items they agree with.

109
Guttman Scales (1944)
  • This pre-ranks statements in order, very similar
    to Thurstone scaling, except here we attempt to
    construct the scale so that if a person agrees
    with item 4 on the scale, they will also agree
    with items 1,2 and 3.
  • When the questions are administered, the items
    are muddled, but each retains a ranking.
  • The respondents score is the sum of the ranks
    associated with the items he or she agreed with.

110
Osgoods (1957)Semantic Differential Scales
  • Respondents are asked to rate an idea or an
    object against a series of opposing adjectives or
    descriptions, for example
  • Using Learnwise
  • Exciting Dull
  • Hard Easy
  • Frustrating
    Stimulating
  • The scale asks respondents to tick the box
    nearest the descriptor that they agree with. Each
    box has a numerical value e.g. 1,2,3 7
  • The respondents score can be portrayed
    graphically, or as a total of numerical values.

111
Questionnaires InterviewsReliability Issues
  • For questionnaire results to be reliable, you
    need closed questions that are precisely framed,
    unambiguous, with response categories that are
    well-defined, exhaustive and exclusive. You need
    to ask the same question in different ways, and
    you need to collate the information into
    statistical summaries and expose the data to
    rigorous statistical testing.
  • You need to use as large a sample as possible,
    and even out any random fluctuations in the data
    by using summary statistics and hypothesis
    testing.

112
Questionnaires InterviewsValidity Issues
  • For questionnaire results to be valid, you need
    response categories which enable the respondent
    to express their views accurately this can best
    be done with open questions. You should take time
    with the respondent and respect the data that
    they provide for you.
  • You need to ensure that the sample is
    representative of the population, and large
    enough for the results to be statistically
    significant.

113
Questionnaire References On-Line
  • http//www.tardis.ed.ac.uk/kate/qmcweb/qcont.htm
  • http//www.leeds.ac.uk/iss/documentation/top/top2.
    pdf
  • http//www.statpac.com/surveys/
  • http//www.fao.org/docrep/W3241E/w3241e05.htmchap
    ter20420questionnaire20design
  • http//www.surveysystem.com/sdesign.htm

114
Questionnaire References (1)(SW Library)
  • Foddy, William. - Constructing questions for
    interviews and questionnaires theory and
    practice. - Cambridge Cambridge University
    Press, 1993. - 0521467330
  • Fowler, Floyd J.. - Improving survey questions
    design and evaluation. - London Sage, 1995. -
    0803945833
  • Frazer, Lorelle. - Questionnaire design and
    administration a practical guide / Lorelle
    Frazer. - Brisbane Wiley, 2000. - 0471342920
  • Gillham, W. E. C., William Edwin Charles, 1936-.
    - Developing a questionnaire. - London
    Continuum, 2000. - (Real world research). -
    0826447953

115
Questionnaire References (2) (SW Library)
  • Oppenheim, A. N., Abraham Naftali, 1924-. -
    Questionnaire design, interviewing and attitude
    measurement / A.N. Oppenheim. - New ed. - London
    Continuum, 2000. - 0826451764
  • Wengraf, Tom. - Qualitative research interviewing
    biographic narrative and semi-structured
    interviews - London SAGE, 2001. - 0803975007
  • Young, Pauline V. - Scientific social surveys and
    research an introduction to the background,. -
    Englewood Cliffs, N.J. Prentice-Hall, 1966. -
    (Prentice-Hall Sociology Series). - m0859969
  • Youngman, Michael Brendan. - Designing and
    analysing questionnaires. - Maidenhead, Berks
    TRC Rediguide. - (Rediguide 12). - m0891542

116
Sampling
  • The question of how to select the respondents for
    the sample is always tricky.
  • The principle behind sampling, is that you should
    ensure that the sample is appropriate in order
    for you to generalise your results to the
    population under study.
  • There are essentially three ways in which this is
    done
  • Random Sampling
  • Quota Sampling
  • Stratified Random Sampling

117
Random Sampling
  • This method requires you to get a list of all the
    population, as near complete as you can find,
    then use some random selection method (such as
    shutting your eyes and stabbing a pen at the
    list, or allocating using random numbers)
  • The idea is that every person in the population
    has an equal chance of ending up in the sample.
  • Most statistical methods assume that you are
    sampling randomly, and it is the only method
    which overall is guaranteed to ensure that
    samples are free from bias, and therefore provide
    validity.

118
Quota Sampling
  • Here, you identify particular sectors of the
    population, such as men, women , those under 21,
    over 21, employed, unemployed etc., and put
    quotas on the number of people in each category.
  • For example, in a quota sample of 12, we might
    have
  • 2 males under 21 2 females under 21
  • 4 males over 21 4 females over 21
  • The purpose here is not to get a sample which
    represents the population in its entirety, but to
    ensure that views from important sections of the
    population are represented.

119
Stratified Random Sampling
  • One of the problems with Random Sampling, is that
    if you take small samples, you can very well end
    up with a biased sample , by chance.
  • In Stratified Random sampling, you would measure
    what proportions of the population lie in each
    category or strata, and select your sample so
    that you get precisely those proportions in those
    strata
  • For example in your population you might have
  • 15 males under 21 10 females under 21
  • 30 males over 21 45 females over 21
  • If you select a sample of 200 you would need
    stratified samples
  • 30 males under 21 20 females under 21
  • 60 males over 21 90 females over 21
  • The allocation of subjects to samples should be
    done randomly.
  • The purpose here is to get a valid sample which
    which represents the population in its entirety.

120
Other Sampling Methods
  • Snowball Sampling This method is used whenever
    you are dealing with a tricky subject, where
    people may be doing devious or illegal activities
    (for example hacking, virus creation etc.). In
    this case, you would use one respondent to
    suggest the name of another who might be willing
    to be interviewed that one would lead to several
    others and so on.
  • Cluster Sampling This method might be used if
    you have a diverse population (such as those
    living in African urban communities). Here you
    would randomly select particular cities, and
    sample within neighbourhoods within those cities.
  • Convenience Sampling This is used mainly for
    investigative research. It relies on the fact
    that people appearing at a particular location or
    time will do so, at random (may not be true).
    We use this fact to stand in one place and
    interview them.

121
Sampling On-Line References
  • This is an easy introduction
  • http//www.csm.uwe.ac.uk/pwhite/SURVEY1/node26.ht
    ml
  • This is more complex and technical
  • http//www.socialresearchmethods.net/kb/external.h
    tm

122
5c.5 Developmental Research
  • This short section describes what is required
    for a project which involves the development of a
    piece of software or an application.

123
Developmental Methods
  • Depending upon the type of developmental work to
    be undertaken, you should engage in some of the
    following
  • An analysis of the current situation.
  • A detailed design for the new piece of software
  • Testing evaluation of the software once it has
    been developed

124
Analysis of the current situation
  • The analysis could involve some or all of the
    following
  • Detailed Systems Analysis of the context of
    application
  • Examination of the theoretical background to
    developmental work of this nature, e.g. what are
    the available technologies, what are the
    theoretical constraints?
  • A Feasibility Study
  • A Cost-Benefit analysis
  • A User Needs Analysis

125
Design for the software
  • The design for the software could involve any of
    the following, as relevant to the particular
    project
  • Detailed plan of the software design structure,
    with Structured English Blocks
  • Storyboards
  • Prototypes
  • Detailed Design Specification e.g. file sizes
    types, screen layouts and resolution etc.

126
Evaluation and Testing (1)
  • Testing might involve
  • Using a code-checker (see e.g. http//w3schools.co
    m for web-pages)
  • Designing a detailed testing procedure where each
    possible input and output is tested methodically
  • Testing the software in-situ

127
Evaluation and Testing (2)
  • Evaluation might involve
  • Observing users working with the software, and
    collecting data on times, reactions etc.
  • Evaluation using experts
  • User focus group sessions, using research
    instruments such as evaluation sheets,
    questionnaires

128
Developmental ResearchReliability Issues
  • For the results of developmental research to be
    reliable, If another developer had been
    developing the software, the solution that you
    have come to should have been reached by that
    developer for that context the solution (and the
    software) should NOT be dependent upon your
    skills and your knowledge, but rather the
    specific requirements of the problem which you
    were trying to solve.
  • To do this you need to ensure that the way that
    the software has been constructed has taken
    account of previous work in this area, and that
    you have made comparisons with currently existing
    software both the existing software in the
    actual context in which you are working, and
    similar software in similar situations.

129
Developmental ResearchValidity Issues
  • For the results of developmental research to be
    valid, you will need to ensure that the piece of
    software resulting from the research exactly
    matches the requirements of the context.
  • The way to do this is to ensure that sufficient
    data is collected about the context prior to
    software design and construction, and that the
    design and product specification are agreed with
    clients and other interested parties,
  • In addition it is crucial that the software is
    tested and evaluated effectively, and account is
    taken of the results of this, in order to ensure
    that the match with user requirements and
    contextual constraints have been satisfied.

130
6. Data Analysis
  • This section of the presentation looks at the
    different types of data available, and how this
    can be analysed

131
Analysis of Research Data
Undertake Literature Review
Select Research Questions
Devise Methodology Research Instruments
Here we examine how the data produced by the
research can be presented effectively and
statistically analysed.
Apply Methods Instruments
Perform Statistical Analysis
Test Hypotheses Draw Conclusions
132
Data Analysis
  • 6a Types Of Data
  • 6b Extracting Data for Analysis
  • 6c Presentation of Data
Write a Comment
User Comments (0)
About PowerShow.com