Developing an Argument Catalogue: What Weve Learned Canadian Network for Knowledge Utilization

1
Developing an Argument CatalogueWhat Weve
Learned Canadian Network for Knowledge
Utilization

• Philip C. Abrami, Robert M. Bernard C. Anne
  Wade
• Concordia University, Montreal, Quebec
• abrami_at_education.concordia.ca

2
What is an Argument Catalogue?

• A systematic compilation of current public,
  policymaking, practitioner and scientific
  opinions and positions on a topic area
• Accesses public information sources (e.g.
  newspapers, policy documents, practitioner
  sources (e.g., trade journals), academic sources
  (e.g., reviews and primary research)
• Identifies overlaps and gaps in multiple
  perspectives.
• Used to focus and structure systematic reviews

3
Overlaps and Gaps in Multiple Perspectives
Gaps
Overlaps
4
Stages in an Argument Catalogue

• Stage 1 Formulating the Purpose and Research
  Question(s)
• Stage 2 Locating and Retrieving Documents
• Stage 3 Including and Excluding Documents
• Stage 4 Creating an Argument Catalogue Codebook
• Stage 5 Coding Documents
• Stage 6 Analyzing and Interpreting the Data
• Stage 7 Disseminating the Results

5
An Example of an Argument Catalogue

• Canadian Council on Learning contract
• State-of-the-field Review of e-learning
• 90-day turn-around time

6
Methodology Locating Documents

• Policy AACE Digital Library, CPI.Q, ERIC,
  FRANCIS, Google, Source OECD, individual websites
• Public (Media) Biblio Branchée, Canadian
  Newsstand, CBCA Current Events, CPI.Q
• Practitioner (Trade) AACE Digital Library, CBCA
  Current Events, CBCA Education, CPI.Q, EBSCO
  Academic Search Premier, ERIC, individual
  websites
• Scholarly articles and reviews AACE Digital
  Library, CBCA Education, Cochrane Library, EPPI,
  ERIC, EBSCO Academic Search Premier, FRANCIS,
  Google Scholar, Medline, ProQuest Digital
  Dissertations and Theses Full-text, PsycINFO

7
Locating Documents Strategy

• Definition E-learning is the development of
  knowledge and skills through the use of
  information and communication technologies (ICTs)
  to support interactions for learninginteractions
  with content, with learning activities and tools,
  and with other people (J. Rossiter, 2002 2005).

8
Locating Documents Search Terms

• Elearning, e-learning, electronic learning
• Technology (or computer) uses AND education
• Distance education, distance learning
• Online instruction, online courses
• Multimedia instruction
• Web-based learning
• Virtual classrooms
• Computer-mediated communication
• Computer-based instruction
• Computer-assisted instruction
• Telemedicine and (learning or education)
• e-health and (learning or education)

9
Search Results Includes

• Includes Of the 1,146 reviewed, 726 were
  selected for inclusion
• Public opinion 235
• Research Studies (Canadian) 152
• Practitioner 131
• Scholarly reviews 120
• Policy Documents 88

10
Search Results Excludes

• Excludes Of the 1,146 reviewed, 420 were
  excluded for the following reasons
• Not e-learning 198
• Related to e-learning in the workplace 43
• Other reasons (specified with a comment) 51
• Primary research conducted outside of Canada 128

11
Breakdown of Includes Sources of Documents
12
Breakdown of Includes CCL Theme Areas
13
Limitations of Searches

• Controlled vocabulary may not be used or may be
  quite broad due to the multidisciplinary nature
  of indices for non-scholarly material
• Abstracts may not be included in the indices thus
  requiring the retrieval of the full document to
  determine relevancy of items
• Policy and trade documents may be scattered
  across different types of retrieval tools
• Access to foreign material may be hampered
• Dates of coverage of the various indices of
  non-scholarly material may be limited (especially
  for newspaper articles)

14
Sample Quantitative Results
15
Sample Qualitative Results

• General public opinion
• One of the major messages found in the general
  public opinion literature is that e-learning is a
  rapidly growing field in education. There is
  speculation that traditional Canadian
  universities will come under increased pressure
  to expand their online offerings because of
  growing enrolments (Online Courses Grow in
  Popularity, 2003, October 18). Online courses,
  and institutions such as Athabasca, are viewed as
  possible solutions to the problems of growing
  enrolments and limited physical space on
  traditional campuses (Summerfield, 2000,
  September 5 Amiot, 2001, Aout 25 Wanless, 2000,
  June 30 Sinnema, 2005, January 29 Rodrigue,
  2001, Novembre 17).

16
Dissemination Document The Knowledge Link
17
Limitations

• Did not weight the primary evidence by sample
  size but instead treated each study equally
• Based primarily on frequency analyses or vote
  counts of impacts without regard to the
  methodological quality of the evidence
• Used a single codebook for each source of
  evidence
• Broad-based analysis lacking depth and detail

18
New Developments

• In-depth analysis of Canadian provincial (10),
  territorial (3), and federal (7)agency policy
  documents on e-learning
• Searched government web sites only
• Harder to be systematic about searching for
  policy documents
• Two coders extracting general policy statements
  for reliability agency contact will verify
  summary
• Did not analyse policy implementation
• Hope the process leads to strategic engagement
  among the agencies regarding the future of
  e-learning in Canada

19
Select References

• Abrami, P.C., Bernard, R.B. Wade, C.A (2006).
  Affecting policy and practice Issues involved
  in developing an argument catalogue. Evidence
  Policy A Journal of Research, Debate and
  Practice, 2(4), 417-437.
• Abrami, P.C., Bernard, R.M., Wade, A., Schmid,
  R.F., Borokhovski, E., Tamin, R., Surkes, M.,
  Lowerison, G., Zhang, D., Nicolaidou, I., Newman,
  S., Wozney, L., Peretiatkowicz (2006). A review
  of E-learning in Canada A rough sketch of the
  evidence, gaps and promising directions. Plus
  commentaries and a rejoinder. Canadian Journal of
  Learning and Technology, 33(1).
• Bernard, R.M. Abrami, P.C. Wade, C.A. (2006). A
  summary of Review of e-Learning in Canada A
  Rough Sketch of the Evidence, Gaps, and Promising
  Directions. Horizons, 9(3), 35-39.

20
Contact Us

• abrami_at_education.concordia.ca
• bernard_at_education.concordia.ca
• wada_at_education.concordia.ca
• Canadian Network for Knowledge Utilization
• Centre for the Study of Learning Performance
• Concordia University
• 1455 deMaisonneuve Blvd. W.
• Montreal, Quebec CANADA H3G 1M8
• 514-848-2424 x2020