Advancing Assessment of Quantitative and Scientific Reasoning - PowerPoint PPT Presentation

Loading...

PPT – Advancing Assessment of Quantitative and Scientific Reasoning PowerPoint presentation | free to download - id: c155f-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Advancing Assessment of Quantitative and Scientific Reasoning

Description:

Center for Assessment and Research Studies (CARS) James Madison University ... Phase II: Validity studies (to be developed and discussed during second faculty ... – PowerPoint PPT presentation

Number of Views:220
Avg rating:3.0/5.0
Slides: 22
Provided by: jmu
Learn more at: http://archive.naspa.org
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Advancing Assessment of Quantitative and Scientific Reasoning


1
Advancing Assessment of Quantitative and
Scientific Reasoning
  • Donna L. Sundre
  • Amy D. Thelk
  • Center for Assessment and Research Studies (CARS)
  • James Madison University
  • www.jmu.edu/assessment/

2
Overview of talk
  • Current NSF Research project
  • History of the test instrument
  • Phase I Results from JMU
  • Phase II Future directions
  • Results from some of our partners
  • Michigan State
  • Truman State
  • Virginia State

3
Current NSF Project
  • 3-year grant funded by National Science
    Foundation Advancing assessment of scientific
    and quantitative reasoning
  • Hersh Benjamin (2002) listed four barriers to
    assessing general education learning outcomes
  • confusion
  • definitional drift
  • lack of adequate measures, and
  • misconception that general education cannot be
    measured
  • This project addresses all of these concerns with
    special emphasis on the dearth of adequate
    measures

4
Objective of NSF project
  • Exploring the psychometric quality and
    generalizability of JMUs Quantitative and
    Scientific Reasoning instruments to institutions
    with diverse missions and serving diverse
    populations.

5
Partner Institutions
  • Virginia State University State-supported
    Historically Black institution
  • Michigan State University State-supported
    Research institution
  • Truman State University State-supported
    Midwestern liberal arts institution
  • St. Marys University (Texas) Independent
    Roman-Catholic Hispanic Serving institution

6
Project phases
  • Phase I First Faculty institute (conducted July
    2007 at JMU) followed by data collection,
    identification of barriers, and reporting of
    results
  • Phase II Validity studies (to be developed and
    discussed during second faculty institute, July
    2008), dissemination of findings and
    institutional reports

7
History of the instrument
  • Natural World test, developed at JMU, currently
    in 9th version
  • Successfully used for assessment of General
    Education program effectiveness in scientific and
    quantitative reasoning
  • Generates two subscores SR and QR
  • Summary of results since 2001
  • Table of Results -- 5 Test Versions.doc

8
Adaptation of an instrument
  • JMU instrument has been carefully scrutinized for
    over 10 years
  • The QR and SR is currently administered at over
    25 institutions across the nation
  • NSF decided to fund this CCLI project to further
    study procedures for adoption and adaptation of
    instruments and assessment models

9
Evaluating the generalizability of the instrument
10
Step 1 Mapping Items to Objectives
  • Relating test items to stated objectives for each
    institution
  • In the past back translation method was used
    (Dawis, 1987) ..\..\JMU\NSF Grant\Truman\Blank
    ObjectiveGrid_truman.doc
  • Participants at the NSF Faculty Institute used a
    new content alignment method that was reported on
    at NCME (Miller, Setzer, Sundre Zeng, 2007)
  • Forms were custom made for each institution
  • Example Content Alignment form.doc

11
Early content validity evidence
  • Results strongly support generalizability of test
    items
  • Truman State 100 of items mapped to their
    objectives
  • Michigan State 98 (1 item not mapped)
  • Virginia State 97 (2 items unmapped)
  • St. Marys 92 (5 items not mapped)
  • Mapping of items alone is not sufficient
  • Balance across objectives must be obtained
  • Teams then created additional items to cover
    identified gaps in content coverage
  • 14 for MSU 11 for St. Marys 10 for Truman
    State 4 for VSU

12
Step 2 Data Collection and Analysis
  • During Fall 2007 semester, test was administered
    to students at 3 of the 4 partner institutions
  • Spring 2008 data collection from students at
    sophomore level or above
  • Results so far
  • Means not given This activity is not intended to
    promote comparison of students across
    institutions
  • At this stage, reliabilities provide the most
    compelling generalizability evidence of course,
    the upcoming validity studies will be informative

13
Score JMU freshmen N1408 SMU freshmen N426 TSU Jrs/Srs N345 VSU N653 MSU N1029
QR a .64 a .63 a .66 a .55 --
SR a .71 a .75 a .72 a .65 --
Total Score NW-9 a .78 a .81 a .79 a .73 a .71
14
Research at JMU
  • Standard Setting to aid in interpretation
  • Validity evidence Instrument aligns with
    curriculum

15
Standard Setting
  • Used Angoff Method to set standards
  • Our process was informal, unique
  • Results look meaningful but well reevaluate as
    we collect more data in upcoming administrations

16
Faculty Objective Standards
17
Validity evidence for instrument and curriculum
at JMU
Variables Pearsons r
Freshman QR9 score AP credits 0.28
Freshman QR9 score DE credits 0.21
Freshman SR9 score AP credits 0.24
Freshman SR9 score DE credits 0.20
18
Validity evidence for instrument and curriculum
at JMU -- 2
Variables Pearsons r
Soph/Jr. NW9 score AP credits 0.16
Soph/Jr. NW9 score DE credits 0.01
19
Phase II studies
  • Samples of Upcoming Studies
  • Correlational Studies Is there a relationship
    between scores on the QR/SR and other
    standardized tests? and other academic
    indicators?
  • Comparison of means or models Is there a
    variation in the level of student achievement
    based upon demographic variables? Is there a
    relationship between scores on the QR/SR and
    declared majors? Can this instrument be used as a
    predictor for success and/or retention for
    specific majors?
  • Qualitative Research Will institutional
    differences be reflected in the results of a
    qualitative interview that accompanies the
    administration of QRSR?

20
References
  • Dawis, R. (1987). Scale construction. Journal of
    Counseling Psychology, 34, 481-489.
  • Hersh, R. H., Benjamin, R. (2002). Assessing
    selected liberal education outcomes A new
    approach. Peer Review, 4 (2/3), 11-15.
  • Miller, B. J., Setzer, C., Sundre, D. L.,
    Zeng, X. (2007, April). Content validity A
    comparison of two methods.  Paper
    presentation to the National Council on
    Measurement in Education. Chicago, IL.

21
Any Questions?
Up next Michigan State University
About PowerShow.com