Tom Weko

1
Nationally Representative Sample Surveys What
Can They Tell Us About Remediation? What Should
They Do Next?

• Tom Weko
• National center for Education Statistics,
• Postsecondary, Adult, and Career Education
• 30 June 2010

2
What Nationally Representative Sample Surveys
Cannot Do

• Research that uses evidence based upon RCT or
  quasi-experimental methods can (potentially)
  provide strong causal inferences about what
  works.
• Nationally representative sample surveys (NRSS)
  are not well-suited to strong causal inferences
  about effects.
• Nonetheless, there are some important
  contributions to be made by NRSS, which Ill
  focus on today.
• They are
• (Obviously), nationally representative data
• That contain student self-reports about things
  that are analytically important, but cannot
  capture through administrative data.

3
But, Theyre Nationally Representative.

• Much of the work on effects of remediation is
  done at an institutional or state level.
• NRSS are the only means by which to address
  questions about the incidence of remedial
  education among a nationally representative
  sample of institutions and students.
• They will remain the only nationally
  representative picture for the foreseeable
  future, since postsecondary SUR systems
• Are still in development (29 SURS in 25 states
  have course-level remediation data)
• Are not inclusive of all institutions (nonprofit,
  for profit)
• Do not follow common data standards
• Are not interoperable.

4
Nationally representative.

• NRSS data about the incidence of remedial
  education are captured by one (or, two) means.
• Self-reported in the student interview. Planned
  for NPSAS2012.
• 1st and 2nd year students Since you completed
  high school, have you taken remedial or
  developmental courses to improve your basic
  skills, such as in mathematics, reading, writing,
  or studying?
•  
• Did you take any remedial or developmental
  courses during the 2011-2012 school year?
•  
• How many remedial or developmental courses did
  you take in the following subjects during the
  2011-2012 school year in Mathematics? Reading?
  Writing?
• Should we attempt to distinguish between course
  attempts and completes in the interview? Include
  study skills?
•  

5
Nationally representative.

• Transcripts.
• NCES secondary longitudinal studies
  (NELS88/2000) have collected and analyzed
  transcripts to examine remedial course taking
  (NCES 2003-394).
• The 2008 Postsecondary Education Transcript
  Study (PETS) has collected and coded 25,190
  transcripts for the 2003/9 BPS study and 16,080
  for the 2008/09 BB study.
• Principal resource is BPS, not BB, since latter
  is completion cohort, and has collected
  transcripts only from BA awarding institution,
  not institution of origin if other than BA
  institution.
• Public use data based upon these transcript data
  will be available in 2011. Released through
  PowerStats (http//nces.ed.gov/datalab/)
• Restricted data based upon these transcripts will
  also be available in 2011.

6
NRSS contain analytically valuable student
reported data

• NRSS contain data elicited in the student
  interview that are missing in administrative
  data.
• Self-reported measures income and parental
  education that can be used to characterize SES,
  so we know who is taking these courses.
• Student information, expectations, and
  evaluations about remedial education. Such as
• Were they expecting this to happen? Before
  enrolling in postsecondary education did they
  seem themselves as having weaknesses in the
  subject(s) for which theyre now taking remedial
  coursework?
• Other student information and expectations that
  may be associated with remediation, such as
  student expectations about the probability of
  transfer or graduation, and expectations about
  wages.

7
What should we do next in NRSS?

• Remedial course taking measured through
  interviews or transcripts has an important
  limitation institutions use different assessment
  instruments and cut scores to assign students to
  the treatment, so were not capturing a
  standardized measure of ability or persistence
  risk.
•  
• Ideally, survey data collections would collect
  which assessments are used, actual assessment
  scores, and cut points for assignment to
  remediation from sampled institutions.
• Can this be done?
• Where on campus do these data reside?
• How long are they maintained?
• How many institutions are using local
  assessments, as opposed to the major industry
  assessments?
• Is it possible to maintain institutional
  participation rates while collecting cut points?
•  

8
What Should We Do Next?

• The National Assessment Governing Board is
  seeking OMB approval for Evaluating Student Need
  for Developmental or Remedial Courses at
  Postsecondary Education Institutions
  http//www.reginfo.gov/public/do/PRAViewICR?ref_nb
  r201006-3098-001
•  120 pilot institutions, July 2010, 1,670 full
  scale 2 and 4 year institutions, November 2010
• NPSAS2012 field test planning to ask
  institutions to provide information on the type
  of placement test administered, and individual
  student scores.
• The payoff? NRSS data would contain
  standardized measures of ability and an
  opportunity to look at what happens when students
  with like assessment scores are exposed to
  different information about their abilities, i.e.
  assigned to remediation or not.