IALS Measurement Validity, Program Impact and Logic Models for Policy Development

1
IALS Measurement Validity, Program Impact and
Logic Models for Policy Development
Stephen Reder Portland State University
I
The Centre for Literacy Fall Institute
Banff, Alberta October 25, 2011
IALS, Its Meaning and Impact for Policy and
Practice
2
The Longitudinal Study of Adult Learning (LSAL)
Portland State University
funded by U.S. Department of
Education and National Institute for Literacy
3
Periods of Participation in Basic Skills Programs
Since Leaving School
4
Self-Study in the LSAL

• Defined as studying on your own to improve your
  reading, writing or math skills or prepare for
  the GED
• Probes distinguished such self-study from
  activity conducted in school or as part of a
  basic skills or GED class
• In-depth qualitative interviews confirmed the
  validity of these self-reports

5
Self-Study by Literacy Proficiency(wave 1)
6
Self-Study Skills Practiced(Wave 3)
7
Self-Study Materials Used(Wave 3)
By Wave 5, computer use for self-study rose to 51
8
Self-Study and Participation
64 have self-studied
self-study only
neither
adult ed only
60 have taken adult ed
self-study adult ed
(between leaving school and Wave 4)
9
Self-Reported Change in Literacy Practices by
Participation Self-Study
10
Percent GED Attainment by Participation
Self-Study
11
Some Implications

• We need a broader construct of participation that
• Includes programs, self-study other learning
  activities
• Sees self-study in a continuum of provision
  strategies along with various types of classroom
  and tutoring programs
• Conceives of learners as actively deploying
  resources as well as programs delivering services

12
Some Implications (cont)

• We need broader programmatic structures that
  support coordinate different modes of
  participation
• Technologies may be particularly helpful here but
  need to provide more than just direct instruction
  (e.g., support goal setting plan development,
  counselling, etc.)

13
Modeling Lifespan Literacy Development

• Program participation and self-study have
  positive, short-term effects on engagement in
  literacy practices
• Engagement in literacy practices over time has
  cumulative effects on the growth of literacy
  proficiency
• This leads to a broader concept of participation
  that involes both programs and self-study
• Self-study bridges between periods of program
  participation and helps broaden persistence of
  attendance into persistence of learning

14
Programs as Practice-BuildingContexts

• Programs that build engagement in authentic
  literacy practices may lead to long-term
  proficiency growth that is not evident in
  short-term pre-post testing

15
Programs as Facilitated LearningContexts

• Programs often blend time in classroom
  instruction with time in learning centers
• There may be a continuum of facilitated learning
  with classroom activities on one end and
  self-study on the other end, with facilitated use
  of learning centers and tutoring in-between

16
Redefining Program, Participation and
Persistence

• Focus on engagement in literacy practices rather
  than on persistence in programs
• Broaden definition of program to include
  facilitated engagement in literacy practices
• Technology has broader role here than just
  delivering distance learning

17
Effects of Instruction

• Strongest evidence of program impact Controlled
  comparison of participants and non-participants
  gains
• Next strongest Covariation of hours of
  instruction with observed gains
• My analysis of CASAS proficiency gains in several
  states NRS data indicates the gains attributable
  to instructional hours are greatly attenuated (if
  present at all) when test experience is controlled

18
Program Participation..

• ..is often fragmented by life circumstances
• ..is better thought of as a busy intersection
  among life and learning trajectories rather than
  as a parking lot
• ..should support students trajectories not just
  their activities in the intersections
• ..is expanded by a learning plan that learners
  can follow and that services can be wrapped
  around

19
Learning Support Systems

• Adults need learning support systems that provide
  portable, personalized learning plans they can
  follow to reach their goals
• Learning support systems include a focus on
  building engagement in literacy practices
• Learning support systems provide flexible blends
  of online and offline local resources to support
  learning
• The technology supports learners directly as they
  follow a learning plan and indirectly by
  coordinating the work of those who support them
• Communities design and implement local learning
  support systems, utilizing the technology to
  facilitate collaboration and information-sharing
  among the various organizations working with
  learners

20
A New Logic Model for Program Impact

• Practice-engagement theory
• Busy intersection vs. parking lot
• Learning support system

21
Research Recommendations

• Improve ways of measuring engagement in literacy
  practices
• Conduct interventions that stimulate increased
  practice-engagement
• Need long-term longitudinal studies with multiple
  repeated literacy outcome measures
• Note that National Reading Panel found little
  evidence indicating that programs which aim to
  increase independent reading in schoolchildren
  lead to increased proficiency
• Examine literacy changes (including proficiency
  loss) in older adults and interventions to
  maintain literacy capabilities in older adults

22
Contact information

• Steve Reder
• Professor, Chair Department of Applied
  Linguistics
• Portland State University
• [phone]
• reders_at_pdx.edu
• For more information and references to the
  research covered in the Powerpoint presentation,
  please see
• Longitudinal Study of Adult Learning
  www.lsal.pdx.edu
• Some Thoughts on IALS Measurement Validity,
  Program Impact, and Logic Models for Policy
  Development www.centreforliteracy.qc.ca/sites/defa
  ult/files/Rederthinkpiece.pdf