Studies of the Effectiveness of ALN: Improving Quantitative Research Methods

1
Studies of the Effectiveness of ALN Improving
Quantitative Research Methods

• Starr Roxanne Hiltz (NJIT)
• J.B. Arbaugh (U. of Wisconsin)
• Lake George, September 2002

2
Background Workshop on Future ALN Research

• Participants and Process
• Plan for the book describes what we know based
  on past research, what we need to know, and the
  methodological techniques (both quantitative and
  qualitative) that are needed to improve our
  knowledge about ALN effectiveness
• This paper is draft chapter on Improving
  Quantitative Research on ALN

3
Two major themes

• A. Issues related to measuring learning (Why
  no significant difference?)
• Measures of student learning and of course
  outcomes
• B. Improving the generalizability of findings in
  ALN studies
• More longitudinal, multi-institutional,
  multi-cultural studies

4
A.1. Methods of Assessing Learning

• If we cannot validly measure learning, then we
  cannot compare the relative effectiveness of
  learning.
• Thus, we will begin with a review of the major
  ways in which learning can be assessed within a
  course, including some problems and limitations
  with each of the possible measures.

5
1.1 Instructor- Assigned Grades (on exams or in
the course)

• Limitations of the timed, proctored, objective
  or short essay exam for validly measuring
  learning, especially in a course that emphasizes
  collaborative learning and deep understanding.
• Instructors tend to curve grades within a
  section. That is, they tend to add points or
  make other adjustments so that each section of
  each course has a grade distribution that looks
  something like a normal curve.

6
Variation The Collaborative Exam

• Students work together on all phases of the exam,
  except individuals answer questions alone.
• NJIT results show the majority of students felt
  they learned throughout the process (including
  making up questions, and grading others'
  answers), that the exams were successful in
  demonstrating what they learned, and it was an
  enjoyable process.
• Probably not possible to implement without ALN
  so not usable for comparative studies

7
1.2 Projects and portfolios

• A "project" is an artifact prepared by an
  individual student or a collaborative group of
  students, that is assigned by the instructor to
  provide an opportunity to apply the knowledge and
  skills covered by texts and lectures
• Portfolios are student-prepared collections of
  documents that evidence understanding of
  important concepts or mastery of key skills,
  requiring students to organize, synthesize, and
  communicate their achievements throughout the
  semester.

8
Projects and portfolios

• May be a more valid measure of learning but time
  consuming to grade
• Need to have a clear grading rubric, so students
  know what is expected and so grades assigned by
  two raters would be similar ( reliability)
• Hardly ever used in comparative studies
• Not a problem ifh focus is on improving
  effectiveness within ALN

9
Participation (number, frequency, length of
comments)

• Automated systems could potentially be developed
  to assign grades for online participation
• (still would not help for comparative studies)

10
A. 2. Methods for Evaluating Course Outcomes

• Whatever the method used by the instructor to
  assess student learning, there are also other
  aspects of the course process and outcomes that
  need to be measured in order to have a complete
  picture of course effectiveness

11
2. Evaluating Course Outcomes

• These include subjective measures of course
  satisfaction by both students and faculty, and
  measures of resource expenditures

12
2.1 Attitudinal Surveys Perceived
Learning/Satisfaction

• Attitudinal surveys, which are most often
  administered as post-course student evaluations,
  can provide valuable information on students'
  perceptions of the course, the discipline, the
  instructor, the materials and tools used, and
  their individual progress and knowledge gains.

13
2.1 Attitudinal Surveys Perceived
Learning/Satisfaction

• Non-response bias is important because
  non-respondents to surveys may very well have
  differing perceptions of the ALN experience,
  thereby making the conclusions drawn from studies
  of low response rates misleading

14
Increasing Response Rates

• Researchers can increase the level of their own
  hands-on involvement in the data collection
  process by means such as personalizing the appeal
  for participation in letters or email, and
  initiating additional mailings to boost response
  rates

15
Increasing Response Rates

• Decrease the inconvenience of completing and
  returning surveys (e.g., online surveys), or
  increase the rewards to students for completing
  them .
• Another validity problem is that the data
  generated through such surveys is self-report
  rather than direct observation. One problem is
  the tendency towards a halo effect

16
Validity problems with surveys

• Attitudinal surveys can provide a summative
  overview of how and/or what students do, think,
  or feel about a given course, but a more detailed
  understanding of students' attitudes requires the
  use of supplementary research methods such as
  individual or group interviews

17
B. Expanding the scope and validity of ALN studies

• ALN research has generally progressed from
  anecdotal experiences with single courses to
  comparisons with classroom learning in single
  course settings, to multi-course and
  multi-discipline ALN studies

18
Research Needs, 1 Multi-Course Studies

• Multi-course studies would provide several
  methodological benefits for ALN research. Two
  direct benefits would be increased external
  validity and statistical power.

19
2. Multi-Institution Studies

• Multi-institution studies would provide the
  opportunity to further generalize research
  findings.
• Presently, nearly all multi-course studies have
  been conducted with a single college or
  university as a research setting. While the
  findings of these studies have certainly been
  fruitful, there are often some rather
  institution-unique characteristics that may limit
  the generalizability of these findings to other
  settings.

20
3. Multi-cultural, multi-national Studies of ALN

• As more schools seek to deliver education via the
  internet to a global audience and create
  collaborative ventures with schools in other
  countries, opportunities for multi-national ALN
  studies will certainly increase in the future

21
4. Need for Longitudinal Studies

• To date, studies of ALN-related changes in
  student, instructor, program, and institutional
  behavior over time have been extremely limited

22
Longitudinal and Multi-national Studies

• Additional research in these areas will greatly
  enhance our understanding of how ALNs evolve and
  provide us with insights on how they may further
  change in the future.

23
Summary and Conclusions

• Improving the scope and validity of quantitative
  methods used to assess learning effectiveness was
  one of the top rated research priorities
  according to the prominent ALN researchers who
  participated in our workshop.

24
Conclusions

• We have examined two significant criteria that
  must be addressed if ALN research is to be useful
  for informing future educational research and
  practice more attention to valid and
  pedagogically sound measurement of ALN learning
  effectiveness, and increased rigor in studies of
  ALNs

25
Conclusions

• While recent advances in multi-course,
  multi-discipline ALN research are encouraging,
  extending this research to study ALNs beyond
  institutional, state, and even national
  boundaries is vital for us to gain understanding
  about the contextual factors that most
  significantly influence ALN effectiveness.