Update on MCAS: Is it Working? Is it Fair? - PowerPoint PPT Presentation

Loading...

PPT – Update on MCAS: Is it Working? Is it Fair? PowerPoint presentation | free to view - id: 519996-ZGM5Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Update on MCAS: Is it Working? Is it Fair?

Description:

Update on MCAS: Is it Working? Is it Fair? Ronald K. Hambleton University of Massachusetts at Amherst EPRA Seminar, November 5, 2005. (revised) – PowerPoint PPT presentation

Number of Views:63
Avg rating:3.0/5.0
Slides: 66
Provided by: margaretl151
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Update on MCAS: Is it Working? Is it Fair?


1
Update on MCAS Is it Working? Is it Fair?
  • Ronald K. Hambleton
  • University of Massachusetts at Amherst
  • EPRA Seminar, November 5, 2005. (revised)

2
Purposes
  • Address some of the misconceptions that exist
    about the MCAS.
  • In addressing the misconceptions, provide some
    insights about MCAS, and answer questions about
    its level of success and its fairness.
  • Share some of my own concerns about MCAS and next
    steps.

3
General Goals of State Testing Programs Like MCAS
  • Provide students, their parents, and teachers
    with feedback on student educational progress in
    relation to state curricula.
  • Compile data for monitoring progress or change in
    student performance over time.
  • Provide data for educational accountability.
    (e.g., NCLB legislation)

4
Characteristics of MCAS Assessments
  • English Language Arts and Mathematics assessments
    at several grades.
  • Science, social studies/history assessments,
    second language proficiency are coming in
    Massachusetts.
  • Multiple-choice (60)and performance tasks (40).
  • Assessments include a core of items for student
    reporting, and other items (for field-testing,
    curriculum evaluation, and linking test forms
    over time)
  • Performance standards set by educators.

5
MCAS is not just about testing! It is about
  • -- substantially increased funding for education
  • --curriculum reform
  • --integrating of curricula, instruction, and
    assessment
  • --improving administrator and teacher training,
    and educational facilities
  • --addressing special needs of students

6
Lets consider next, six common criticisms of
MCAS.
7
1. State tests encourage teachers to teach to
the test and this narrows the curriculum taught.
  • This is potentially a valid concernProblematic
    with NRTs10 to 20 coverage of curricula, MC
    items only. Same skills and items are assessed
    each year. Here, teaching narrowly to the
    skills and content of the test improves test
    scores but not learning of the broader curricula.
    But MCAS assessments are not NRTs!!

8
1. State tests encourage teachers to teach to
the test and this narrows the curriculum
taught.
  • MCAS Assessments are representative of the
    curricula and new items are used each year. What
    does teaching to the test mean when the tests
    are a sample of the curricula? Teach the
    curricula!
  • Consider the next set of displays85 or more of
    MCAS curricula are assessed in every three year
    cycle.

9
Comparison of Percent of Learning Standards
Assessed in Math at Grades 4, 6, 8, and 10 from
2001 to 2004. (about 40/grade)
10
Percent of learning standards assessed in
Mathematics at grade 4, 6, 8 and 10 in the time
periods, 2001 to 2003, and 2002 to 2004.
11
(No Transcript)
12
  • In sum, no justification for narrowing the
    curricula The assessments are representative of
    the curricula, and over three year periods, over
    85 of learning standards are being assessed.
    (Results at all grades and subjects are equally
    good.) Teaching to the test/assessment means
    teaching the curricula!
  • Other states, too (e.g., Minnesota) have found
    that when tests and curricula are aligned,
    teachers are considerably more supportive of
    educational assessments. Teachers need to see
    these alignment results in Massachusetts!

13
2. Important decisions about students should
not turn on one test.
  • AERA, APA, and NCME Test Standards highlight
    importance of measurement errors, and the
    undesirability of a single test driving an
    important decision.
  • Reality State tests (e.g., grade 10 ELA and
    Math) are not the only requirements for
    studentsEnglish, mathematics, science, history
    credits at HS level are required, as well as
    regular attendance.

14
2. Important decisions about students should not
turn on one test.
  • Students have five chances to pass grade 10 ELA
    and math assessments during their high school
    years.
  • Appeals process (for students close to the
    passing score, and who are attending school
    regularly, and taking the assessments).
  • Math assessment is available in Spanish too (to
    reduce bias).

15
2. Important decisions about students should not
turn on one test.
  • DOE is expecting schools to be doing their own
    assessments too (using multiple methods, such as
    projects, portfolios, work samples, classroom
    tests, etc.)I think one might question high
    school grading practices at grades 11 and 12 if a
    grade 10 test was a major block for graduation.
  • In sum, the criticism does not have merit.

16
3. State assessments are full of flawed and/or
biased test items.
  • Item writing is not a perfect science and
    mistakes will be made.
  • Massachusetts releases all operational items on
    the web-site shortly after their use. Find the
    flawed items if you can? (I cant and I have
    seriously looked.) This is a remarkable
    situationfew states release itemsexcellent for
    instructional purposes and critics. If critics
    think items are flawed, report them.

17
3. State assessments are full of flawed and/or
biased test items.
  • Process of preparing items in Massachusetts is
    state-of-the art qualified and culturally
    diverse item writers content and bias reviews by
    committees, department, and contractors field
    testing study of statistical evidence for bias
    and care in item selection (optimal statistically
    and content valid).

18
3. State assessments are full of flawed and/or
biased test items.
  • UMass has looked at over 1000 items over years,
    grades, and tests, and found little statistical
    evidence for gender and racial bias.
  • I just dont see the merit of this criticism, and
    I have studied these tests to find flaws and
    biases and cant (but for a few items in science).

19
4. Student testing takes up too much time and
money.
  • Quality tests are expensive, and require student
    time. (Reliability of scores needs to be high.)
    (six hours in some gradesgrades 4 and 10)
  • In Massachusetts, for example, all students at
    grades 3, 4, 6, 7, 8, and 10 are tested for some
    subjects.

20
4. Student testing takes up too much time and
money. (Cont.)
  • 4 to 6 hours/per student or about 0.5 of
    instructional time per year. (one day of 180!)
  • 7.0 billion on education, 25 million for
    assessment/year, 20.00/student. 0.3 of
    education budget, or 1 of every 300 dollars spent
    on MCAS assessments! Seems obvious that the
    amount of time and cost of assessments is not out
    of line with value.

21
Changes in timing or scheduling of the
testreaction to criticisms in 1998.
  • Administer test in short periods.
  • Administer at a time of day that takes into
    account the students medical needs or learning
    style.
  • Time of testing varies by grades, but takes less
    than one day (total) of 180 days of school year,
    not all grades assessed, and diagnostic results
    from students and groups can be used to improve
    instructional practices!

22
One Example Using Item Analysis Results at the
School Level (reproduced with permission of MDOE)
Students Performing At the Proficient Level
Your school
23
5. Passing scores are set too high.
  • Too often judgment of pass rates is made based on
    failure rates.
  • Look at the process used by the states, look for
    validity evidence.
  • Who is setting the passing scores and what method
    are they using?
  • What is the evidence to support performance
    standards that are set too high in Massachusetts?
    It doesnt exist, in my judgment.

24
5. Passing scores are set too high.
  • Typically, passing scores are set by educators,
    school administrators and sometimes parents and
    local persons are included too. In Mass,
    teachers dominated. (52 of panelists)
  • Critics need to study the procedures used in
    setting passing scores and validity evidence.
  • As an expert on this topic, I can tell you that
    the state used exemplary procedures.

25
5. Passing scores are set too high.
  • Test scores are placed on a new reporting scale
    with scores from 200 to 280. 220 is passing.
  • In 2005, more than 75 of grade 10 students
    passed both ELA and math assessments (first
    time), and pass rates over 80 for each
    assessment (first time takers).
  • I dont see merit in the criticism.

26
6. There is little or no evidence that MCAS is
producing results.
  • Internal evidence (sample)
  • --At the grade 10 level, pass rates have been
    steadily increasing.
  • --Research evidence by Learning Innovations
    (2000) 90 of schools indicated changes in
    curricula changes influenced by test results
    instruction influenced by MCAS results in over
    70 of teachers.

27
6. There is little or no evidence that MCAS is
producing results.
  • External evidence
  • --State received very positive reviews about the
    MCAS curricula from Achieve, Inc. (a national
    review group)--among the best in country.
  • --NAEP scores are up since 1992 for white, Black,
    and Hispanic students.
  • --SAT scores are up, and more students taking the
    SAT.

28
  • NAEP 2005
  • Massachusetts and National Results
  • Percentages at NAEP Achievement Levels
  • Mathematics Grade 4
  • Reading Grade 4

29
  • Mathematics Grade 4
  • Percentage at NAEP Achievement Levels

Source Massachusetts Snapshot Report 2005 US
DOE, IES, NCES
30
  • Reading Grade 4
  • Percentage at NAEP Achievement Levels

Source Massachusetts Snapshot Report 2005 US
DOE, IES, NCES
31
1994-2004 Massachusetts Mean SAT Scores Combined
Verbal Math
MA Nation
32
Personal Concerns
  • Drop out rates have increased, especially for
    inner-city students. But how much? Why? What
    can be done if true?
  • Retention rates at the ninth grade are up. How
    much? Why? What can be done?
  • Consequential validity studies are needed.
    Intended and unintended outcomesboth positive
    and negative need to be identified, and addressed.

33
Personal Concerns
  • Funding of schools. Is it sufficient? Are we
    spending the money on the right items and in the
    appropriate amountsteachers, special programs,
    school facilities, etc.? (Assessment results
    provide clues, at least, to problem areas.)

34
Conclusions
  • I am encouraged by educational reform in
    Massachusettsmany positive signs funding,
    curricula, assessments, concern for students who
    need special assistance, etc.
  • Internal and external validity evidence is very
    encouraging.
  • Importance problems remainnotably achievement
    gap, and funding issues.

35
Conclusions
  • I am troubled by the misconceptions that are so
    widely held about the MCAS. They interfere with
    effective implementation.
  • Would like to see everyone get behind educational
    reform, and make it work for more students.
    Continue with the strengths and address problems.
  • --Compile substantial validity evidence, then
    make the necessary changes, with the goal to make
    education in Massachusetts meet the needs of all
    students.

36
Follow-up reading
  • R. P. Phelps. (Ed.). (2005). Defending
    standardized testing. Mahwah, NJ Lawrence
    Erlbaum Publishers.

37
  • Please contact me at rkh_at_educ.umass.edu for a
    copy of the slides, or to forward your questions
    and reactions.

38
Some extra slides. Not used in the presentation
because of limited time.
39
(No Transcript)
40
State approach to minimizing drop-outs
  • Provide clear understanding to students about
    what is needed.
  • Improve students classroom curricula and
    instruction.
  • Offer after-school and summer programs.
  • Find new role for community colleges to meet
    student needs.
  • Do research to identify reasons for drop-outs and
    then react if possible.

41
7. Testing accommodations are not provided to
students with disabilities.
  • Federal legislation is very clear on the need for
    states to provide test accommodations to students
    who need them. (ADA, IDEA legislation)
  • Validity of scores is threatened.
  • State provides a large set of accommodations.

42
Long List of Available Accommodations
  • About 20 accommodations organized into four main
    categories(a) changes in timing, (b) changes in
    setting, (c) changes in administration, and (d)
    changes in responding.

43
b. Changes in test setting
  • Administer to a small group or private room
  • Administer individually
  • Administer in a carrel
  • Administer with the student wearing noise buffers
  • Administer with the administrator facing student

44
c. Changes in test administration
  • Using magnifying equipment or enlargement devices
  • Clarifies instruction
  • Using large-print or Braille editions
  • Using tracking items
  • Using amplification equipment
  • Translating into American Sign Language

45
d. Changes in how the student responds to test
questions
  • Answers dictated
  • Answers recorded

46
8. State tests must be flawed because failure
rates are high and better students do go onto
jobs and colleges.
  • Actually failure rates at the grade 10 level are
    not high. (80 pass both tests on first chance)
  • NAEP results are not that far out-of-line with
    state results in New England. in fact, results
    are close
  • Too many colleges must offer basic reading and
    math courses.

47
8. State tests must be flawed because failure
rates are high and better students go onto jobs
and colleges.
  • Internationally, we are about middle of the pack.
    In one of the recent studies, we were right
    there with Latvia and New Zealand, and trailing
    Korea, Singapore, and many other industrial
    countries.

48
9. Test items are biased against minorities.
  • Another excellent validity concern, but the
    evidence is not supportive of the charge in
    Massachusetts.
  • We have analyzed available 1998, 2000, 2001 grade
    4, 8, 10, ELA, math, science, history
    Male-Female Black-White Hispanic-Black.

49
Conditional P-Value Plot of Uniform DIF
(SDIF0.135, UDIF0.136)
50
Conditional P-Value Plot of Non-Uniform DIF
(SDIF0.060, UDIF0.029)
51
Grade 10 Mathematics DIF Item Plot
52
DIF Indices for a Mathematics Test (M-F)
53
Mathematics Test Items (M-F) Organized by Content
54
Even prior to statistical review, judgmental
reviews take place
  • Assessment development committee
  • Bias review committee
  • Department staff
  • External content area experts
  • Item writers themselves

55
The real discrimination is more likely in the
educational system not the assessments.
  • Discrimination does exist in an educational
    system that has historically moved students
    forward despite poor educational performance,
    awarded high school diplomas, and sent them on to
    minimum wage jobs. If high schools wont stop
    practice, then the state needs to intervene.

56
10. Gains in achievement are most likely due to
improvements in testing practices only.
  • Many possible reasons for gain includingstudents
    are learning to take tests (not all bad),
    aligning of instruction to tests (not bad if
    tests measure curriculum frameworks), cheating
    (little evidence so far), and holding back
    students/drop-outs.

57
Consider retention argument to explain
achievement growth in Massachusetts retention
rate increased 25! (see Walt Haney)
  • Reality It went from 4 at ninth grade to 5.
    Increase 25.
  • With 60,000 students, 600 students affected, pass
    and fail rates would be affected by about 1
    only! This is not the explanation for growth.

58
10. Gains in achievement are most likely due to
improvements in testing practices only. (Cont.)
  • Also, it is possible that teachers and students
    are working harder, teachers are focused on the
    curricula, and are teaching more effectively.

59
10. Gains in achievement are most likely due to
improvements in testing practices only. (Cont.)
  • Research evidence by Learning Innovations (2000)
    90 of schools indicated changes in curricula
    changes influenced by test results instruction
    influenced by MCAS results in over 70 of
    teachers.

60
11. Not everything that should be tested is
included.
  • Definitely truebut over time, what can be tested
    should be tested. And, schools have the
    responsibility to pick up the rest! They can use
    work samples, portfolios, more performance tasks,
    etc. There are course grades too.

61
12. Special education students should not be
included.
  • Federal laws (ADA, IDEA) require that every
    possible effort to be made to include all
    students in the assessments. The policy is know
    as full inclusion. President Bushs no child
    left behind is another example of federal
    initiatives.

62
Conclusions
  • From a technical perspective, many state tests,
    especially recent ones, are quite sound.
  • Technical methods for test development, equating,
    assessing reliability and validity, setting
    standards, etc. are very much in place and
    suitable.

63
Conclusions (Cont.)
  • Major shortcoming of many state testing programs
    (1) too little emphasis on diagnostic testing,
    and (2) too little emphasis on efforts to
    evaluate program impact.
  • Impact on students such as drop-outs, retentions,
    attitudes, going to college, etc. needed

64
Conclusions (Cont.)
  • Impact on teachers such as improvements in their
    qualifications, changes in instructional
    practices, attitudes about teaching, etc. needs
    study
  • Impact on school administrators such as the need
    for new forms of leadership, and new demands on
    time, needs study.

65
Conclusions (Cont.)
  • Testing methods and curricula frameworks are in
    place (and can be modified as appropriate).
  • My hope would be that educators try and make the
    current program of educational reform
    workevaluate as we go, and revise accordingly.
    A data-based approach make sense for effective
    educational reform.
About PowerShow.com