IdMRC Social Research Methods Autumn LectureWorkshop Series

1
IdMRC Social Research MethodsAutumn
Lecture-Workshop Series
2
Design Research Methods Observation and
Coding

• Janet McDonnell

3
Note

• This collection of slides supports a lecture on
  the topic of observation and coding.
• The slides in this set are not intended to serve
  independently of the lecture as a stand-alone
  introduction to, or overview of, the subject.
• The lecture includes the use of video and other
  visual material excised from this sub-set of
  slides which is solely intended to support note
  taking at the lecture.
• Janet McDonnell

4
Outline

• software tools
• CASE STUDY from the categorical to the
  conjectural
• Knowledge claims from nature to narrative
• Traceability and reliability
• What is the data from observation
• Approaches to research using observation data
• CASE STUDY single dataset, multiple approaches
  to analysis
• Pointers for quality

5
Software Tools - Lets get them out of the way

• Behavioural Research recording behaviour, eye
  tracking, facial expression and other types of
  behaviour recognition, synchronising with
  transcripts and support for coding, etc.
• www.mangold-international.com (Interact)
• www.tracksys.co.uk (Observer XT)
• Qualitative Data Analysis content analysis of
  texts from transcripts, interview data, etc.
  linking in images and video clips, qualitative
  data base management, collaboration
• www.qsrinternational.com (Nvivo)
• www.atlasti.com (ATLAS-ti)
• Quantitative data analysis
• Stats packages, graphing and diagramming tools
• Transcription aids

6
Anatomy of a small scale enquiry

• An investigation into the exchange of ideas and
  information between an architect and building
  users in the early stages of a building
  (re)design project before the design brief or any
  drawings have been produced.
• We look at the type of information users
  exchange.
• We are interested in what influences the
  information exchanges - context of the meetings
  
• and
• - the conversational strategies of the
  architect.
• Classifications were determined in advance of
  coding the data.

7
Data and method

• Participant observation interaction as it occurs
  in natural setting
• Data consists of transcripts acquired from
  audio-taped meetings
• Focus on manifest content
• Segment transcripts into units of analysis
• Apply predetermined codes
• Inter coder issues
• Data analysis and data interpretation
• Construct an account of the topic
• Based on P. Medway (2000) Writing and Designing
  in Architectural Education in
• A.Pare (ed) Transitions writing in academic and
  workplace settings, Hampton Press, pp 89-129.

8
Conversation summaries
9
Information contributions
10
Codes for information contributions
11
shelving reachable from the flooreasy to
clean surfaces light and quiet study areas
a place where it feels safe to think a
link between our past and our future
12
Information contributions
13
It was noticeable that some users were competent
at discussing contributions in a more informed,
technical manner than others. There were
differences between the users understanding of
the structural properties of buildings and their
range of abilities to articulate spatial
preferences for the design of the workspace.
Compare U3 my wish is I suppose to take
out this joint wall and incorporate this store
and this area into a larger area with U1 if
I had my wish then I would basically um take this
wall down Although conversation 3 was shorter
than the others, U3s ability to converse in a
designerly language made it easier to share a
common understanding of the properties of the
space.
14
Types of information contribution
15
The contribution of information with a symbolic
meaning was infrequent. Symbolic references only
cropped up in conversation 4 all contributed by
the deputy head teacher rather than the primary
user of the space. We note also that the symbolic
meaning discussed concerned what it meant to be
part of the school and only once was the symbolic
meaning of the space being designed mentioned
strictly. The three instances are D this is
regarded as a privilege to give them a kind of
base so that they can start acting can I say more
acceptable, normal circumstances in their
reaction and inter-reactions with other
people and D thats why we still insist on
them being in school uniform so theyre still
part of the school to stamp identity U1 er yeah
the whole idea is that theyre to be
re-integrated into school and D we use this
facility as an escape to do course work in the
privacy of here as opposed to the pressures in
the school
16
Claims, evidence and truth

• what is provably true
• logico-deductive reasoning, exhaustive search
• what is probably true
• statistical reasoning
• what is plausibly true
• the evidence of arguments that are sincere and
  will convince a reasonable audience

17
Knowledge claims from research

• Provable and Probable
• Positivist natural sciences experimental
  reproducibility falsifiable hypotheses
• Plausible
• Interpretivist cultural and social sciences
  plausibility traceability recoverability
  narratives providing understanding
• Emancipatory
• Critical theorist social sciences
  emancipatory force

18
Transparency in enquiry
methodology
framework of ideas
area of research concern

• Declared or shared framework, methodology,
  methods allow process by which results are
  obtained to be recovered by others
• The process of enquiry may lead to contributions
  to knowledge in each aspect (framework,
  methodology, research area)
• Sharing of the meaning-making addresses
  subjectivity objections a route to agreement

19
Inter coder reliability

• Context decisions about what is required to
  adequately serve the claims and evidence
• Kappa coefficient is a measure of agreement
  between coders on the classification of
  qualitative data
• Measures agreement in using a defined coding
  scheme in a prescribed manner.
• Kappa takes into account the potential for chance
  agreement.
• Does not take account of degrees of disagreement.
• Is a good test of whether categories are
  sufficiently distinct from each other and
  sufficiently clearly defined for more than one
  coder to code reliably using them. So can be used
  to test identification of units of analysis and
  application of codes.
• J Carletta (1996) Computational Linguistics
  22(2) 249-254 Assessing agreement on
  classification tasks the kappa statistic.

20
Extract from a transcript
21
Transcription conventions
  pause of one second ( two seconds, and
so on) /.\ crude indicators of simultaneous
speech /\ NO emphatic stress GOOGLE name
s of people, organisations, and so
on material omitted - incomplete or cut
off utterance ( ) unclear utterance (over
there) unclear utterance, transcribers best
guess within brackets points at
drawings transcribers comments laughs para
linguistics  
                         
For a more comprehensive scheme refer to Gail
Jeffersons notation described in J Atkinson and
J Heritage (eds) Structures of Social Action
(1984) CUP, pp ix-xvi
22
Ways of approaching research using observation
data

• Starting with a template
• e.g. a model or a theory, classifications
  determined in advance that are imposed on the
  data
• Starting with the data
• e.g. looking for emergent patterns, evolving
  classification to cover the phenomena observed
  (coding scheme not inevitable consequence)
• Flexible research agenda vs. rigid research
  agenda
• e.g. starting with a scheme but being open to the
  unclassifiable (and to new classes) developing
  hypotheses to test based on indications of
  possible patterns, causations

23
Anatomy of a large collaboration DTRS7

• Common Dataset
• Multi-angle video recordings, plans, drawings,
  sketches, flip-charts, transcripts from meetings
  from two authentic design projects
• Architectural Design Project Data
• Two meetings 8 months apart between architect and
  clients/building users
• Brief to design a crematorium with chapel and
  related provision including offices, vestry,
  waiting rooms, parking, landscaping
• Engineering / Product Design Project Data
• Two meetings 3 days apart among engineers and
  other specialists from the same company
  multi-disciplinary groups
• Brief to develop novel product ideas for sort of
  digital pen to exploit a new technology
• 24 different studies on themes relevant to
  research into design thinking

24
How the data was tackled

• Styles of approaching the data
• Starting with a template e.g. a model or a
  theory
• Starting with the data
• Flexible agenda vs. rigid agenda
• Themes
• Understanding Process
• Values in Designing
• Aspects of Design Cognition
• Design Process Models
• Language, Discourse, and Gesture
• Designing Contexts
• Objects, References, and Representations

25
Types of analysis imposition of a model or
theory

• Focus of attention uses of analogy in designing
• Finding a previously unreported use of
  analogy, namely function-finding in creative
  design
• Ball and Christensen, 2009
• Focus of attention object references during
  designing
• Finding object references in meetings between
  co- workers are directed effectively to draw
  attention efficiently to features essential
  to the comparison Stacey et al., 2009
• Focus of attention language of appraisals
• Finding positive and negative appraisals have
  different effects on the focus of attention
  and knowledge generation during design
• Dong et al., 2009

26
Types of analysis fine grained analysis of the
data

• Focus of attention gestures and individual
  turns at talking
• Finding reveals fundamental distinctions
  between view of designing as goal-oriented
  and as goal-directed Glock, 2009
• Focus of attention designing through
  talk-in-interaction
• Finding design concept has a special status
  less negotiable the preserve of the
  designated design expert
• Luck, 2009
• Focus of attention social order the rules
  of interaction
• Finding adherence to interactional norms
  interferes with the rules of brainstorming
• Matthews, 2009

27
Types of analysis testing the limitations of
prior findings or models

• Focus of attention consideration of context
  during designing
• Finding support for claims that experienced
  designers explore broad context before close
  context is considered in depth
• Atman et al., 2009
• Focus of attention a design teams development
  of shared notions about the task and the
  teams operation (indicated by verbal
  communication) based on a prior model of the
  development of sharedness in teams
• Findings some expected findings predicted by
  the model some unexpected findings (not
  predicted by prior model) only some of
  these accounted for by particularities of the
  data Badke-Schaub et al., 2009

28
Quality of work

• For in-vivo studies err on the side of over
  collection of data
• And/or iterate over observation/data collection
  and analysis
• and dont make excuses
• Coding invest in adequate granularity (beware
  of findings which are artifacts of the coding or
  analysis)
• An underused question what else could account
  for the findings?
• Consider the arguments, claims and evidence
  explicitly at epistemological level
• Choose research methods that will provide the
  supporting evidence you need (e.g. inter coder
  agreement if that is critical)
• Choose language with care consider the
  baggage e.g. hypothesis, subjects

29
Further information

• The small scale enquiry was conducted with Dr
  Rachael Luck of Reading University and is
  published in Design Studies 27 (2006) pp 141-166.
• DTRS7 work is published as a book AboutDesigning
  J.McDonnell and P.Lloyd (eds), 2009 and parts of
  it in two journal special issues Design Studies
  30 (2) and CoDesign 5 (1) both in March 2009.
• Janet McDonnell
• Central Saint Martins
• Southampton Row
• London WC1B 4AP
• j.mcdonnell_at_csm.arts.ac.uk
• www.csm.arts.ac.uk/janet-mcdonnell.htm