JExam - A Method to Measure Outcomes Assessment - PowerPoint PPT Presentation

Loading...

PPT – JExam - A Method to Measure Outcomes Assessment PowerPoint presentation | free to download - id: 24d92e-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

JExam - A Method to Measure Outcomes Assessment

Description:

University of Georgia Chemistry Department. JExam - A Method to Measure Outcomes Assessment ... Plot for 1st test of the second semester of General Chemistry. ... – PowerPoint PPT presentation

Number of Views:19
Avg rating:3.0/5.0

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: JExam - A Method to Measure Outcomes Assessment


1
JExam - A Method to Measure Outcomes Assessment
  • Charles H. Atwood, Kimberly D. Schurmeier,
  • and Carrie G. Shepler
  • Chemistry Department
  • University of Georgia
  • Athens, GA 30602
  • Presented to
  • Academic Affairs Faculty Symposium
  • Unicoi State Park, Helen GA
  • March 31, 2007

2
Outline of Presentation
  • JExam Computerized Testing Program
  • Item Response Theory (IRT)
  • Using IRT to measure testing effectiveness
  • Outcomes Assessment
  • Conclusion

3
Item Response Theory (IRT)
  • Item Response Theory (IRT)
  • More modern psychometric analysis tool than CTT
  • Works well for large sample sizes (gt 200)
  • Iterative process which fits the students
    responses to a mathematical formula

4
Item Response Theory (IRT)
  • We use the program Bilog-MG3 to fit our data.
  • Bilog-MG3 assigns to every test item values for
    a, b, and c and displays them on an item
    characteristic curve
  • Plot of the probability of a student with a given
    IRT ability vs. probability that the student
    answered the question

5
Item Response Theory (IRT)
6
Item Response Theory (IRT)
  • Less than 5 minutes later an item response list
    is generated.
  • Our raw data
  • 00000000 991099919919919919919909919909919
    00000001 919919199199199199199199199199199
    00000002 919919199199199199199199199199199
    00000003 919909199199199199199199199199199
    00000004 919919199099099099199199199199099
    00000005 919919199199099099199199199199199
    00000006 919919199199199199199099199199199
    00000007 919909199199099199199199199199099
    00000008 919919199199099199199199199099199
    00000009 919919199199099099199099099199199
    00000010 919919199199199099199199199199199
    00000011 919919199199199099199199199199099
    00000012 919919199199099199199199199199199
    00000013 919919199199199199199199199099199
    00000014 919919199199199199199199199199199
    00000015 919909199199199199199099199099099
    00000016 990099909919909909919909919909919
    00000017 990099919919909909919919919919919
    00000018 991099919909909909909909909909909
    00000019 991099919919909919909919919919909
    00000020 991099919919909909919909919919919
    00000021 991099919919919919919919919919919
    00000022 991199909919919919919919919919919
    00000023 990099909909919909909909909909909
    00000024 991199919909909919919919919909919

7
Item Response Theory (IRT)
  • We enter the raw data into Bilog-MG 3.0 along
    with a few parameters.
  • Less then 5 minutes later we have our Item
    Characteristic Curves for every question on the
    test.

8
Item Response Theory (IRT)
  • Item Information Curve for every test question

9
Item Response Theory (IRT)
  • C-D discriminator example

10
Item Response Theory (IRT)
  • B-C discriminator example

11
Item Response Theory (IRT)
  • A-B discriminator example

12
Using IRT to measure testing effectiveness
  • Can tell us whether or not our tests are
    effectively assessing students at every ability
    level.
  • Test information curve
  • Prior to IRT

13
Using IRT to measure testing effectiveness
  • Test information curve
  • After IRT

14
Using IRT to measure testing effectiveness
  • Generate a plot of student ability versus test
    score for every test during the semester.
  • Plot for 1st test of the second semester of
    General Chemistry.

15
Using IRT to measure testing effectiveness
  • When student ability vs. test scores is averaged
    over the last 6 years
  • Grading scale that appears to be consistent for
    the entire year and independent of student
    population
  • Possible absolute grading scale

16
Using IRT to measure testing effectiveness
  • If we generate a plot of the number of students
    that have given ability on a test
  • Get the regular bell-shaped curve
  • Plot for Spring semester 2006, 1st test

17
Using IRT to measure testing effectiveness
  • Compare that to this plot
  • Spring Semester 2006, 1st test

18
Outcomes Assessment
  • After Spring Semester 2006, we went and looked at
    the test questions which had the highest
    discrimination and ability levels.
  • From this analysis we discovered that there are
    several topics which cause the students the most
    difficulty for the entire year.

19
Outcomes Assessment
  • Understanding the structure of ionic compounds
  • Unit conversion problems
  • Particularly converting from volume to area or
    height
  • Molecular polarity
  • Intermolecular forces
  • Understanding quantum numbers
  • Distinguishing the terms strong, weak,
    concentrated and dilute
  • Inorganic nomenclature

20
Outcomes Assessment
  • Understanding the structure of ionic compounds is
    crucial to student performance in 1211/1212
  • This concept is essentially a gatekeeper.

21
Outcomes Assessment
Fall 2005 Test 1 Question What is the correct
name of this ionic compound? Al(NO3)3 aluminum
nitrate How many ions are present in one formula
unit of the compound shown above? 4
Ability at 0.336 Discriminates between C and D
students
22
Outcomes Assessment
  • Since understanding ions is so difficult and so
    crucial to success, we decided to tackle it
    first.
  • Prior to start of Fall semester 2006
  • Teaching faculty meeting
  • Discussed difficult topics
  • Stressed that we must emphasize ions in lecture

23
Outcomes Assessment
  • Fall 2006 Test 1
  • Question
  • What is the correct name of this ionic
    compound? Al(NO3)3
  • aluminum nitrate
  • How many ions are present in one formula unit of
    the compound shown above?
  • 4

24
Outcomes Assessment
  • Look at the Gaussian Fit to Ability Scores for
    several years
  • Plot for Fall Semester 2004, 1st test

25
Outcomes Assessment
  • Compare that to this plot
  • Fall Semester 2005, 1st test

26
Outcomes Assessment
  • Compare that to this plot
  • Fall Semester 2006, 1st test

27
Outcomes Assessment
  • Second Test Comparisons
  • Fall Semester 2004, 2nd test

28
Outcomes Assessment
  • Second Test Comparisons
  • Fall Semester 2005, 2nd test

29
Outcomes Assessment
  • Second test comparisons
  • Fall Semester 2006, 2nd test

30
Outcomes Assessment
  • Third test comparisons
  • Fall Semester 2004, 3rd test

31
Outcomes Assessment
  • Third test comparisons
  • Fall Semester 2005, 3rd test

32
Outcomes Assessment
  • Third test comparisons
  • Fall Semester 2006, 3rd test

33
Outcomes Assessment
  • Final Exam comparisons
  • Fall Semester 2004, Final Exam

34
Outcomes Assessment
  • Final Exam comparisons
  • Fall Semester 2005, Final Exam

35
Outcomes Assessment
  • Final Exam comparisons
  • Fall Semester 2006, Final Exam

36
Conclusion
  • Item Response Theory is a more effective method
    of assessing student ability.
  • We can design and implement test questions that
    discriminate student abilities at all levels.
  • Appropriate interventions can measurably improve
    overall student abilities.

37
Acknowledgements
  • Bob Scott and John Stickney
  • Department Chairs
  • Gary Lautenschlager
  • Professor of Psychology and IRT expert
  • UGAs PRISM project.
  • This material is based on work supported by the
    National Science Foundation under Grant No.
    EHR-0314953.
About PowerShow.com