Title: Studies of the Effectiveness of ALN: Improving Quantitative Research Methods
1Studies of the Effectiveness of ALN Improving
Quantitative Research Methods
- Starr Roxanne Hiltz (NJIT)
- J.B. Arbaugh (U. of Wisconsin)
- Lake George, September 2002
2Background Workshop on Future ALN Research
- Participants and Process
- Plan for the book describes what we know based
on past research, what we need to know, and the
methodological techniques (both quantitative and
qualitative) that are needed to improve our
knowledge about ALN effectiveness - This paper is draft chapter on Improving
Quantitative Research on ALN
3Two major themes
- A. Issues related to measuring learning (Why
no significant difference?) - Measures of student learning and of course
outcomes - B. Improving the generalizability of findings in
ALN studies - More longitudinal, multi-institutional,
multi-cultural studies
4A.1. Methods of Assessing Learning
- If we cannot validly measure learning, then we
cannot compare the relative effectiveness of
learning. - Thus, we will begin with a review of the major
ways in which learning can be assessed within a
course, including some problems and limitations
with each of the possible measures.
51.1 Instructor- Assigned Grades (on exams or in
the course)
- Limitations of the timed, proctored, objective
or short essay exam for validly measuring
learning, especially in a course that emphasizes
collaborative learning and deep understanding. - Instructors tend to curve grades within a
section. That is, they tend to add points or
make other adjustments so that each section of
each course has a grade distribution that looks
something like a normal curve.
6Variation The Collaborative Exam
- Students work together on all phases of the exam,
except individuals answer questions alone. - NJIT results show the majority of students felt
they learned throughout the process (including
making up questions, and grading others'
answers), that the exams were successful in
demonstrating what they learned, and it was an
enjoyable process. - Probably not possible to implement without ALN
so not usable for comparative studies
71.2 Projects and portfolios
- A "project" is an artifact prepared by an
individual student or a collaborative group of
students, that is assigned by the instructor to
provide an opportunity to apply the knowledge and
skills covered by texts and lectures - Portfolios are student-prepared collections of
documents that evidence understanding of
important concepts or mastery of key skills,
requiring students to organize, synthesize, and
communicate their achievements throughout the
semester.
8Projects and portfolios
- May be a more valid measure of learning but time
consuming to grade - Need to have a clear grading rubric, so students
know what is expected and so grades assigned by
two raters would be similar ( reliability) - Hardly ever used in comparative studies
- Not a problem ifh focus is on improving
effectiveness within ALN
9Participation (number, frequency, length of
comments)
- Automated systems could potentially be developed
to assign grades for online participation - (still would not help for comparative studies)
10A. 2. Methods for Evaluating Course Outcomes
- Whatever the method used by the instructor to
assess student learning, there are also other
aspects of the course process and outcomes that
need to be measured in order to have a complete
picture of course effectiveness
112. Evaluating Course Outcomes
- These include subjective measures of course
satisfaction by both students and faculty, and
measures of resource expenditures
122.1 Attitudinal Surveys Perceived
Learning/Satisfaction
- Attitudinal surveys, which are most often
administered as post-course student evaluations,
can provide valuable information on students'
perceptions of the course, the discipline, the
instructor, the materials and tools used, and
their individual progress and knowledge gains.
132.1 Attitudinal Surveys Perceived
Learning/Satisfaction
- Non-response bias is important because
non-respondents to surveys may very well have
differing perceptions of the ALN experience,
thereby making the conclusions drawn from studies
of low response rates misleading
14Increasing Response Rates
- Researchers can increase the level of their own
hands-on involvement in the data collection
process by means such as personalizing the appeal
for participation in letters or email, and
initiating additional mailings to boost response
rates
15Increasing Response Rates
- Decrease the inconvenience of completing and
returning surveys (e.g., online surveys), or
increase the rewards to students for completing
them . - Another validity problem is that the data
generated through such surveys is self-report
rather than direct observation. One problem is
the tendency towards a halo effect
16Validity problems with surveys
- Attitudinal surveys can provide a summative
overview of how and/or what students do, think,
or feel about a given course, but a more detailed
understanding of students' attitudes requires the
use of supplementary research methods such as
individual or group interviews
17B. Expanding the scope and validity of ALN studies
- ALN research has generally progressed from
anecdotal experiences with single courses to
comparisons with classroom learning in single
course settings, to multi-course and
multi-discipline ALN studies
18Research Needs, 1 Multi-Course Studies
- Multi-course studies would provide several
methodological benefits for ALN research. Two
direct benefits would be increased external
validity and statistical power.
192. Multi-Institution Studies
- Multi-institution studies would provide the
opportunity to further generalize research
findings. - Presently, nearly all multi-course studies have
been conducted with a single college or
university as a research setting. While the
findings of these studies have certainly been
fruitful, there are often some rather
institution-unique characteristics that may limit
the generalizability of these findings to other
settings.
203. Multi-cultural, multi-national Studies of ALN
- As more schools seek to deliver education via the
internet to a global audience and create
collaborative ventures with schools in other
countries, opportunities for multi-national ALN
studies will certainly increase in the future
214. Need for Longitudinal Studies
- To date, studies of ALN-related changes in
student, instructor, program, and institutional
behavior over time have been extremely limited
22Longitudinal and Multi-national Studies
- Additional research in these areas will greatly
enhance our understanding of how ALNs evolve and
provide us with insights on how they may further
change in the future.
23Summary and Conclusions
- Improving the scope and validity of quantitative
methods used to assess learning effectiveness was
one of the top rated research priorities
according to the prominent ALN researchers who
participated in our workshop.
24Conclusions
- We have examined two significant criteria that
must be addressed if ALN research is to be useful
for informing future educational research and
practice more attention to valid and
pedagogically sound measurement of ALN learning
effectiveness, and increased rigor in studies of
ALNs
25Conclusions
- While recent advances in multi-course,
multi-discipline ALN research are encouraging,
extending this research to study ALNs beyond
institutional, state, and even national
boundaries is vital for us to gain understanding
about the contextual factors that most
significantly influence ALN effectiveness.