Title: Research Methods in Distance Education: Design Based Research
1Research Methods in Distance Education Design
Based Research
- Terry Anderson
- PhD Seminar
- Leicester UK
- Feb. 2012
2My Focus The context of Distance Education
Implementation
- Disruptive innovation (Christensen, 2008)
simpler, not wanted by main stream customers - Rapid gains in functionality
- Cheaper
- Adaptive
- Moving from peripheral to mainstream (blended and
online for full time students)
3Research Paradigms
4Research Paradigms
5Research Paradigms
- Quantitative discovery of the laws that govern
behavior - Qualitative understandings from an insider
perspective - Critical Investigate and expose the power
relationships - Design-based interventions, interactions and
their effect in multiple contexts
6Paradigm 1Quantitative Research
- employs a scientific discourse derived from the
epistemologies of positivism and realism.
7- those who are seeking the strict way of truth
should not trouble themselves about any object
concerning which they cannot have a certainty
equal to arithmetic or geometrical demonstration - (Rene Descartes)
- Inordinate support and faith in randomized
controlled studies
8Quantitative Example 1 CMC Content Analysis
- Anderson, Garrison, Rourke 1997-2003
- http//communitiesofinquiry.com - 9 papers
reviewing results focusing on reliable ,
quantitative analysis - Identified ways to measure teaching, social and
cognitive presence - Most reliable methods are beyond current time
constraints of busy teachers - Questions of validity
- Serves as basic research as grounding for AI
methods and major survey work of the future - Serves as qualitative heuristic for teachers and
course designers
9Quantitative Meta-Analysis
- Aggregates many effect sizes creating large Ns
more powerful results. - Ungerleider and Burns (2003)
- Systematic review of effectiveness and efficiency
of Online education versus Face to face - The type of interventions studied were
extraordinary diverse only criteria was a
comparison group - Only 10 of the 25 studies included in the
in-depth review were not seriously flawed, a
sobering statistic given the constraints that
went into selecting them for the review.
10- 1928-2008
- distance delivery modes from correspondence
schools, radio, television, video, and now
e-learning - when the course materials and teaching
methodology are held constant, there are no
significant differences (NSD) in learner outcomes
http//www.nosignificantdifference.org/ Slide
from Tom Reeves
11Ungerleider, C., Burns, T. (2003). A systematic
review of the effectiveness and efficiency of
networked ICT in education. Ottawa Industry
Canada.
12Is DE Better than Classroom Instruction?Project
1 2000 2004
- Question How does distance education compare to
classroom instruction? (inclusive dates
1985-2002) - Total number of effect sizes k 232
- Measures Achievement, Attitudes and Retention
(opposite of drop-out) - Divided into Asynchronous and Synchronous DE
Bernard, R. M., Abrami, P. C., Lou, Y.
Borokhovski, E., Wade, A., Wozney, L., Wallet,
P.A., Fiset, M., Huang, B. (2004). How does
distance education compare to classroom
instruction? A meta-analysis of the empirical
literature. Review of Educational Research,
74(3), 379-439.
13Quantitative Research Summary
- Can be useful especially when fine tuning well
established practice - Provides incremental gains in knowledge, not
revolutionary ones - The need to control context often makes results
of little value to practicing professionals - In times of rapid change too early quantitative
testing may mask beneficial positive capacity - Will we ever be able to afford blind reviewed,
random assignment studies?
14Paradigm 2 Qualitative Paradigm
- Many different varieties
- Generally answer the question why rather then
what, when or how much? - Presents special challenges in distributed
contexts due to distance between participants and
researchers - Currently most common type of DE research (Rourke
Szabo, 2002)
15Qualitative Example
- Dearnley (2003) Student support in open learning
Sustaining the Process - Practicing Nurses, weekly F2F tutorial sessions
- Phenomenological study using grounded theory
discourse
16Core category to emerge was Finding the
professional voice
Dearnley and Matthew (2003 and 2004)
17Qualitative example 2
- Mann, S. (2003) A personal inquiry into an
experience of adult learning on-line.
Instructional Science 31 - Conclusions
- The need to facilitate the presentation of
learner and teacher identities in such a way that
takes account of the loss of the normal channel - The need to make explicit the development of
operating norms and conventions - reduced communicative media there is the
potential for greater misunderstanding - The need to consider ways in which the developing
learning community can be open to the other of
uncertainty, ambiguity and difference
183rd ParadigmCritical Research
- Asks who gains in power?
- David Nobles critique of digital diploma Mills
most prominent Canadian example - Are profits generated from user generated content
or OERs exploitative? - Confronting the net changes everything mantra
of many social software proponents. - Who is being excluded from online world?
19See Norm Friesens
Friesen, N. (2009) Re-thinking e-learning
research foundations, methods, and practices.
Peter Lang Publishers
20(No Transcript)
21- Why does Facebook own all the content that we
supply? - Does the power of the net further marginalize the
non-connected? - Why did the One Laptop Per Child project fail? Or
did it?
22Do Either Qualitative or Quantitative Methods
Meet Real Needs of Practicing Distance Educators?
23But what type of research has most effect on
practice?
- Kennedy (1999) - teachers rate relevance and
value of results from each of major paradigms. - No consistent results teachers are not a
homogeneous group of consumers but they do find
research of value - The studies that teachers found to be most
persuasive, most relevant, and most influential
to their thinking were all studies that addressed
the relationship between teaching and learning.
24But what type of research has most effect on
Practice?
- The findings from this study cast doubt on
virtually every argument for the superiority of
any particular research genre, whether the
criterion for superiority is persuasiveness,
relevance, or ability to influence practitioners
thinking. Kennedy, (1999)
25Design-Based Research
- Developed from frustration of the lack of impact
of educational research on educational systems.
264th ParadigmDesign-Based Research
- Related to engineering and architectural research
- Focuses on the design, construction,
implementation and adoption of a learning
initiative in an authentic context - Related to Development Research
- Closest educators have to a home grown research
methodology
27Design-Based Research Model
Phase 2
Phase 4
Phase 1
Phase 3
Analysis of Practical Problems by Research and
Practitioners in Collaboration
Development of Solutions Informed by Existin
Design Principles and Technological Innovations
Reflection to Produce Design Principles and
Enhance Solutions Implementation
Iterative Cycles of Testing and Refinement of
Solutions in Practice
Refinement of Problems, Solutions, Methods, and
Design Principles (Reeves, 2006, p. 59)
Reeves, T. C. (2006). Design research from the
technology perspective. In J. V. Akker, K.
Gravemeijer, S. McKenney, N. Nieveen (Eds.),
Educational design research (pp. 86-109). London
Routledge.
28Design-Based Research Studies
- iterative,
- process focused,
- interventionist,
- collaborative,
- multileveled,
- utility oriented,
- theory driven and generative
- (Shavelson et al, 2003)
29Critical characteristics of design experiments
- According to Reeves (20008), Ann Brown (1992)
and Alan Collins (1992) - addressing complex problems in real contexts in
collaboration with practitioners, - integrating known and hypothetical design
principles with technological affordances to
render plausible solutions to these complex
problems, and - conducting rigorous and reflective inquiry to
test and refine innovative learning environments
as well as to define new design-principles.
30Design-based research
- Methodology developed by educators for educators
- Developed from American pragmatism Dewey
(Anderson, 2005) - Recent Theme Issues
- The Journal of the Instructional Sciences, (13,
1, 2004), - Educational Researcher (32, 1, 2003) and
- Educational Psychologist (39, 4, 2004)
- See bibliography at http//cider.athabascau.ca/CID
ERSIGs/DesignBasedSIG/ - My article at www.cjlt.ca/abstracts.html
31Anderson, T., Shattuck, J. (2012). Design-Based
Research A Decade of Progress in Education
Research? Educational Researcher, 41(Jan/Feb.),
16-25. Retrieved from http//edr.sagepub.com/cont
ent/41/1/7.full.pdfhtml.
32Anderson, T., Shattuck, J. (2012). Design-Based
Research A Decade of Progress in Education
Research? Educational Researcher, 41(Jan/Feb.),
16-25. Retrieved from http//edr.sagepub.com/cont
ent/41/1/7.full.pdfhtml.
33Design Tradition
- Learning and productivity are the results of the
designs (the structures) of complex systems of
people, environments, technology, beliefs and
texts New London Group 2000 - DBR opens the door for teachers, researchers and
learners to become designers, not merely
consumers, bosses or observers .
34Integrative Learning Design(Bannan-Ritland, 2003)
35- design-based research enables the creation and
study of learning conditions that are presumed
productive but are not well understood in
practice, and the generation of findings often
overlooked or obscured when focusing exclusively
on the summative effects of an intervention Wang
Hannafin, 2003
36- Iterative because
- Innovation is not restricted to the prior design
of an artifact, but continues as artifacts are
implemented and used - Implementations are inevitably unfinished
(Stewart and Williams (2005) - intertwined goals of (1) designing learning
environments and (2) developing theories of
learning (DBRC, 2003)
37Design Based research and the Science of
Complexity
- Complexity theory studies the emergence of order
in multifaceted, changing and previously
unordered contexts - This emerging order becomes the focus of iterate
interventions and evaluations - Order emerges at the edge of chaos in response
to rapid change, and failure of previous
organization models
38DBR ExamplesCall Centres At Athabasca
Answer 80 of student inquiries Savings of over
100,000 /year Anderson, T. (2005). Design-based
research and its application to a call center
innovation in distance education. Canadian
Journal of Learning and Technology, 31(2), 69-84
39Design Based research in Action
- Phase 1 Exploration surveys, talking to faculty
and tutors, investigating open source tools,
setting research questions - Phase 2. Building the intervention Elgg through
two versions and 85 plugins (on going)
40Design Based research in Practice
- Athabasca Landing
- Elgg based
- Started in 2008
- 3500 users
- Unpaced
- Paced Courses
- Informal Learning
- Staff and alumni networking
- Problem of critical mass
41- Phase 3 Evaluation Before and after surveys
see - Anderson, T., Poelhuber, B., McKerlich, R.
(2010). Self Paced Learners Meet Social Software.
Online Journal of Distance Education
Administration - Dr students Use of past student archives
- Ongoing iterations and development of tools
- Phase 4 Design Principles
- Development of design principles/patterns Nets
and Sets Dron, J., Anderson, T. (2009). How the
crowd can teach. In S. Hatzipanagos S.
Warburton (Eds.), Handbook of Research on Social
Software and Developing Community Ontologies.
42Network Tool Set (example)
Text
Text
Stepanyan, Mather Payne, 2007
43Access Controls in Elgg
44- Need to study usability, scalability and
innovation adoption within bureaucratic systems - Allow knowledge tools to evolve in natural
context through supportive nourishment of staff
45Conclusion
- Education research is grossly under-resourced to
meet the magnitude of opportunity and demand - Paradigm wars are unproductive
- Design-based research offers a promising new
research design model - It can be used for Doctoral dissertations see
- Herrington, J., McKenney, S., Reeves, T.,
Oliver, R. (2007). Design-based research and
doctoral students Guidelines for preparing a
dissertation proposal.
46Your comments and questions most welcomed!
- Terry Anderson terrya_at_athabascau.ca
- Blog terrya.edublogs.org
- Twitter terguy