Research Methods in Distance Education: Design Based Research

1
Research Methods in Distance Education Design
Based Research

• Terry Anderson
• PhD Seminar
• Leicester UK
• Feb. 2012

2
My Focus The context of Distance Education
Implementation

• Disruptive innovation (Christensen, 2008)
  simpler, not wanted by main stream customers
• Rapid gains in functionality
• Cheaper
• Adaptive
• Moving from peripheral to mainstream (blended and
  online for full time students)

3
Research Paradigms
4
Research Paradigms
5
Research Paradigms

• Quantitative discovery of the laws that govern
  behavior
• Qualitative understandings from an insider
  perspective
• Critical Investigate and expose the power
  relationships
• Design-based interventions, interactions and
  their effect in multiple contexts

6
Paradigm 1Quantitative Research

• employs a scientific discourse derived from the
  epistemologies of positivism and realism.

7

• those who are seeking the strict way of truth
  should not trouble themselves about any object
  concerning which they cannot have a certainty
  equal to arithmetic or geometrical demonstration
• (Rene Descartes)
• Inordinate support and faith in randomized
  controlled studies

8
Quantitative Example 1 CMC Content Analysis

• Anderson, Garrison, Rourke 1997-2003
• http//communitiesofinquiry.com - 9 papers
  reviewing results focusing on reliable ,
  quantitative analysis
• Identified ways to measure teaching, social and
  cognitive presence
• Most reliable methods are beyond current time
  constraints of busy teachers
• Questions of validity
• Serves as basic research as grounding for AI
  methods and major survey work of the future
• Serves as qualitative heuristic for teachers and
  course designers

9
Quantitative Meta-Analysis

• Aggregates many effect sizes creating large Ns
  more powerful results.
• Ungerleider and Burns (2003)
• Systematic review of effectiveness and efficiency
  of Online education versus Face to face
• The type of interventions studied were
  extraordinary diverse only criteria was a
  comparison group
• Only 10 of the 25 studies included in the
  in-depth review were not seriously flawed, a
  sobering statistic given the constraints that
  went into selecting them for the review.

10

• 1928-2008
• distance delivery modes from correspondence
  schools, radio, television, video, and now
  e-learning
• when the course materials and teaching
  methodology are held constant, there are no
  significant differences (NSD) in learner outcomes

http//www.nosignificantdifference.org/ Slide
from Tom Reeves
11
Ungerleider, C., Burns, T. (2003). A systematic
review of the effectiveness and efficiency of
networked ICT in education. Ottawa Industry
Canada.
12
Is DE Better than Classroom Instruction?Project
1 2000 2004

• Question How does distance education compare to
  classroom instruction? (inclusive dates
  1985-2002)
• Total number of effect sizes k 232
• Measures Achievement, Attitudes and Retention
  (opposite of drop-out)
• Divided into Asynchronous and Synchronous DE

Bernard, R. M., Abrami, P. C., Lou, Y.
Borokhovski, E., Wade, A., Wozney, L., Wallet,
P.A., Fiset, M., Huang, B. (2004). How does
distance education compare to classroom
instruction? A meta-analysis of the empirical
literature. Review of Educational Research,
74(3), 379-439.
13
Quantitative Research Summary

• Can be useful especially when fine tuning well
  established practice
• Provides incremental gains in knowledge, not
  revolutionary ones
• The need to control context often makes results
  of little value to practicing professionals
• In times of rapid change too early quantitative
  testing may mask beneficial positive capacity
• Will we ever be able to afford blind reviewed,
  random assignment studies?

14
Paradigm 2 Qualitative Paradigm

• Many different varieties
• Generally answer the question why rather then
  what, when or how much?
• Presents special challenges in distributed
  contexts due to distance between participants and
  researchers
• Currently most common type of DE research (Rourke
  Szabo, 2002)

15
Qualitative Example

• Dearnley (2003) Student support in open learning
  Sustaining the Process
• Practicing Nurses, weekly F2F tutorial sessions
• Phenomenological study using grounded theory
  discourse

16
Core category to emerge was Finding the
professional voice
Dearnley and Matthew (2003 and 2004)
17
Qualitative example 2

• Mann, S. (2003) A personal inquiry into an
  experience of adult learning on-line.
  Instructional Science 31
• Conclusions
• The need to facilitate the presentation of
  learner and teacher identities in such a way that
  takes account of the loss of the normal channel
• The need to make explicit the development of
  operating norms and conventions
• reduced communicative media there is the
  potential for greater misunderstanding
• The need to consider ways in which the developing
  learning community can be open to the other of
  uncertainty, ambiguity and difference

18
3rd ParadigmCritical Research

• Asks who gains in power?
• David Nobles critique of digital diploma Mills
  most prominent Canadian example
• Are profits generated from user generated content
  or OERs exploitative?
• Confronting the net changes everything mantra
  of many social software proponents.
• Who is being excluded from online world?

19
See Norm Friesens
Friesen, N. (2009) Re-thinking e-learning
research foundations, methods, and practices.
Peter Lang Publishers
20
(No Transcript)
21

• Why does Facebook own all the content that we
  supply?
• Does the power of the net further marginalize the
  non-connected?
• Why did the One Laptop Per Child project fail? Or
  did it?

22
Do Either Qualitative or Quantitative Methods
Meet Real Needs of Practicing Distance Educators?
23
But what type of research has most effect on
practice?

• Kennedy (1999) - teachers rate relevance and
  value of results from each of major paradigms.
• No consistent results teachers are not a
  homogeneous group of consumers but they do find
  research of value
• The studies that teachers found to be most
  persuasive, most relevant, and most influential
  to their thinking were all studies that addressed
  the relationship between teaching and learning.

24
But what type of research has most effect on
Practice?

• The findings from this study cast doubt on
  virtually every argument for the superiority of
  any particular research genre, whether the
  criterion for superiority is persuasiveness,
  relevance, or ability to influence practitioners
  thinking. Kennedy, (1999)

25
Design-Based Research

• Developed from frustration of the lack of impact
  of educational research on educational systems.

26
4th ParadigmDesign-Based Research

• Related to engineering and architectural research
• Focuses on the design, construction,
  implementation and adoption of a learning
  initiative in an authentic context
• Related to Development Research
• Closest educators have to a home grown research
  methodology

27
Design-Based Research Model
Phase 2
Phase 4
Phase 1
Phase 3
Analysis of Practical Problems by Research and
Practitioners in Collaboration
Development of Solutions Informed by Existin
Design Principles and Technological Innovations
Reflection to Produce Design Principles and
Enhance Solutions Implementation
Iterative Cycles of Testing and Refinement of
Solutions in Practice
Refinement of Problems, Solutions, Methods, and
Design Principles (Reeves, 2006, p. 59)
Reeves, T. C. (2006). Design research from the
technology perspective. In J. V. Akker, K.
Gravemeijer, S. McKenney, N. Nieveen (Eds.),
Educational design research (pp. 86-109). London
Routledge.
28
Design-Based Research Studies

• iterative,
• process focused,
• interventionist,
• collaborative,
• multileveled,
• utility oriented,
• theory driven and generative
• (Shavelson et al, 2003)

29
Critical characteristics of design experiments

• According to Reeves (20008), Ann Brown (1992)
  and Alan Collins (1992)
• addressing complex problems in real contexts in
  collaboration with practitioners,
• integrating known and hypothetical design
  principles with technological affordances to
  render plausible solutions to these complex
  problems, and
• conducting rigorous and reflective inquiry to
  test and refine innovative learning environments
  as well as to define new design-principles.

30
Design-based research

• Methodology developed by educators for educators
• Developed from American pragmatism Dewey
  (Anderson, 2005)
• Recent Theme Issues
• The Journal of the Instructional Sciences, (13,
  1, 2004),
• Educational Researcher (32, 1, 2003) and
• Educational Psychologist (39, 4, 2004)
• See bibliography at http//cider.athabascau.ca/CID
  ERSIGs/DesignBasedSIG/
• My article at www.cjlt.ca/abstracts.html

31
Anderson, T., Shattuck, J. (2012). Design-Based
Research A Decade of Progress in Education
Research? Educational Researcher, 41(Jan/Feb.),
16-25. Retrieved from http//edr.sagepub.com/cont
ent/41/1/7.full.pdfhtml.
32
Anderson, T., Shattuck, J. (2012). Design-Based
Research A Decade of Progress in Education
Research? Educational Researcher, 41(Jan/Feb.),
16-25. Retrieved from http//edr.sagepub.com/cont
ent/41/1/7.full.pdfhtml.
33
Design Tradition

• Learning and productivity are the results of the
  designs (the structures) of complex systems of
  people, environments, technology, beliefs and
  texts New London Group 2000
• DBR opens the door for teachers, researchers and
  learners to become designers, not merely
  consumers, bosses or observers .

34
Integrative Learning Design(Bannan-Ritland, 2003)
35

• design-based research enables the creation and
  study of learning conditions that are presumed
  productive but are not well understood in
  practice, and the generation of findings often
  overlooked or obscured when focusing exclusively
  on the summative effects of an intervention Wang
  Hannafin, 2003

36

• Iterative because
• Innovation is not restricted to the prior design
  of an artifact, but continues as artifacts are
  implemented and used
• Implementations are inevitably unfinished
  (Stewart and Williams (2005)
• intertwined goals of (1) designing learning
  environments and (2) developing theories of
  learning (DBRC, 2003)

37
Design Based research and the Science of
Complexity

• Complexity theory studies the emergence of order
  in multifaceted, changing and previously
  unordered contexts
• This emerging order becomes the focus of iterate
  interventions and evaluations
• Order emerges at the edge of chaos in response
  to rapid change, and failure of previous
  organization models

38
DBR ExamplesCall Centres At Athabasca

Answer 80 of student inquiries Savings of over
100,000 /year Anderson, T. (2005). Design-based
research and its application to a call center
innovation in distance education. Canadian
Journal of Learning and Technology, 31(2), 69-84
39
Design Based research in Action

• Phase 1 Exploration surveys, talking to faculty
  and tutors, investigating open source tools,
  setting research questions
• Phase 2. Building the intervention Elgg through
  two versions and 85 plugins (on going)

40
Design Based research in Practice

• Athabasca Landing
• Elgg based
• Started in 2008
• 3500 users
• Unpaced
• Paced Courses
• Informal Learning
• Staff and alumni networking
• Problem of critical mass

41

• Phase 3 Evaluation Before and after surveys
  see
• Anderson, T., Poelhuber, B., McKerlich, R.
  (2010). Self Paced Learners Meet Social Software.
  Online Journal of Distance Education
  Administration
• Dr students Use of past student archives
• Ongoing iterations and development of tools
• Phase 4 Design Principles
• Development of design principles/patterns Nets
  and Sets Dron, J., Anderson, T. (2009). How the
  crowd can teach. In S. Hatzipanagos S.
  Warburton (Eds.), Handbook of Research on Social
  Software and Developing Community Ontologies.

42
Network Tool Set (example)
Text
Text
Stepanyan, Mather Payne, 2007
43
Access Controls in Elgg
44

• Need to study usability, scalability and
  innovation adoption within bureaucratic systems
• Allow knowledge tools to evolve in natural
  context through supportive nourishment of staff

45
Conclusion

• Education research is grossly under-resourced to
  meet the magnitude of opportunity and demand
• Paradigm wars are unproductive
• Design-based research offers a promising new
  research design model
• It can be used for Doctoral dissertations see
• Herrington, J., McKenney, S., Reeves, T.,
  Oliver, R. (2007). Design-based research and
  doctoral students Guidelines for preparing a
  dissertation proposal.

46

Your comments and questions most welcomed!

• Terry Anderson terrya_at_athabascau.ca
• Blog terrya.edublogs.org
• Twitter terguy