The following strategies are based on a theoretical approach to standardized test deconstruction still in its design phase. This PowerPoint is not finished. Some slides may be incomplete.

1
The following strategies are based on a
theoretical approach to standardized test
deconstruction still in its design phase. This
PowerPoint is not finished. Some slides may be
incomplete.
2
Cracking the Code

• A Strategy for Student Achievement by
  Deconstructing Standardized Tests
• from the Inside-Out!

Curt Von Wooster M.A., M.A., Ed.D., NBCT
3
Central Theory of Deconstruction

• If you can't measure it, you can't manage it.
• Peter Drucker
• (19092005) - a writer, management consultant,
  and self-described social ecologist.
• .In 1959, Drucker coined the term knowledge
  worker.

4
Knowledge Worker Defined

• Knowledge Worker (also referred to as
  intellectual worker or brain worker) is someone
  who is employed due to his or her knowledge of a
  subject matter, rather than their ability to
  perform manual labor.
• We educators are all
• Knowledge Workers!
• As such, we all need to think about testing in a
  more holistic way.
• We need to work smarter when confronting testing
  and not harder!

5
Why do our students not do well on tests? And the
answer is.

• They do not READ on a regular basis.
• As we all know, studies show that students who
  read regularly have increased test scores.
• If we can improve their reading and vocabulary
  comprehension of the actual test, they will do
  better on the test.

6
I am convinced

• .that our low test score problem is NOT due to
  lax or poor instruction on our part as educators.
• Instead, I believe that the our low scores are
  due to poor comprehension of the material and of
  the tests themselves.
• If students cant read the tests, they will do
  poorly on the tests!

7
We need an improved school wide strategy!

• Deconstruct the Tests!

8
The truth is..

• We can and do effectively teach students all day
  long- all year, but proper instruction alone will
  not solve the problem.
• If there was an easier way or perhaps shortcuts
  to helping kids retain the information thereby
  increasing test scores, wouldn't we be foolish
  not to use it.
• If our school funding is so tied to test scores
  and we risk losing it if we do not increase our
  AYP/API standing, then why not try an additional
  approach to test preparation.!

9
Strategy-Based Achievement

• Previously the school focused on Goals and a
  Vision for achieving higher scores on tests.(ex
  Do your best on the test!)
• Simple strategies were sometimes focused
  school-wide. Often too little and too late. (ex
  Re-teaching and reviews)
• Past Professional Development days were geared
  toward success on the test - usually only a month
  before the actual May assessments.

10
Post vs. Pre Analytic Strategy

• Post-Analytic Strategies were focused on doing
  well on the test by getting students to learn how
  to take the test (the external approach) based on
  past test scores from previous years. analyzing
  after the test?
• Pre-Analytic Strategies focus on what comprises
  the test and how it was designed (the internal
  approach). This is where deconstructing the tests
  is crucial. We need to know where the testing
  problems are before they occur! analyze before
  the test!
• We need to use BOTH effectively in unison!

11
We need a three fold approach
12
Metrics defined .

• Test design relies on metrics.
• Metrics/metrical design- set of ways of
  quantitatively and periodically measuring,
  assessing, controlling or selecting a person,
  process, event, or institution, along with the
  procedures.. to carry out measurements and the
  procedures for the interpretation of the
  assessment in the light of previous or comparable
  assessments.
• Wordy, huh!? Instead lets simplify it.

13
Psychometric Design.

• Standardized tests do not design themselves. They
  are not teacher designed.
• Each state directs the contracted testing
  companies with their individual needs, and
  requests that tests be designed accordingly. The
  testing company then goes to work designing.
• Each test is designed by teams of
  Psychometricians.
• Psychometricians are highly trained individuals
  who are skilled in mathematics, statistical
  analysis, educational theory and psychology.
• If we take even a small amount of time to
  understand the test design process and
  psychometrics, then we will be ahead of the game.
  
• Think of the test as the enemy!
• Understanding the enemy is our goal, in order
  that we can learn its weaknesses and formulate a
  more thoughtful approach to attacking the test.

14
To understand the test we must understand who
makes it.

• The same company who makes the state tests
  Educational Testing Service or ETS licenses some
  of their questions to be used on LAUSD periodic
  assessments.
• Just as any product that emanates from its
  manufacturer has certain attributes and inherent
  design functions, ETS tests have a certain rhythm
  and pattern to them. We need to review every
  aspect of these tests to identify certain ETS
  design considerations.
• The same goes for other testing companies.

15
Design Philosophy.defined.

• Example Ford makes automotives.
• All Fords have certain design similarities.
• As a result, we can make certain assumptions
  about what is important to Fords design
  philosophy.
• Powerful engines
• Sleek designs
• Fuel efficiency
• Strong
• Dependable
• A certain Ford attitude meant to make consumers
  want to drive their vehicles

• We can look at the state standardized tests and
  LAUSD periodic assessments the same way!

16
Believe it or not!?

• Nothing on any standardized assessment is by
  chance. Modern metrical test design is viewed as
  both science and an art.
• Nothing, from the layout and language to the
  graphics, italicization and bolding of certain
  words or letters is by accident.
• Not even the use of certain syllables and the
  frequency of certain words, themes or answers is
  an accident. Even the number of questions on a
  page is by design.
• Review released questions from past tests and see
  for yourself.

17
Playing the game..

• There are only three ways to play any game
• 1. Play to Win
• 2. Play to Lose
• 3. Play to break even or to a draw
• Vegas makes money because they understand these
  rules.
• We need to more effectively play to WIN
• rather than talk and try to win!

18
In shortThese tests are a game!

• Any game can be won with the right strategies.
• All games include inherent design flaws or
  weaknesses.
• If we choose to view them as a game we can beat
  them!
• Whether we choose to admit it or not, we tend to
  teach to the test. We might as well try to beat
  the test!
• We can WIN the testing game with the right
  strategies and approach!

19
Beautiful minds? Game theory and test design.

• Remember this movie. It is the story of
  psychiatric patient and mathematics professor
  John Nash who helped form the science of Game
  Theory.
• By using simple game theory principles
  (predicting the probability of certain outcomes)
  we can predict the subject matter on the test and
  also student responses.
• Not a mathematician or insane, no problem.
• But more on that later

20
And now on with the show!!.

• Now that you have learned some crucial
  information about standardized testing and their
  design, lets put this information into practice.
• Remember, this school should continue to do all
  of the things we already do to properly
  instruct!!!!!!!!!!!!
• We just need to add to our arsenal of
  instructional weapons, some enhanced strategies
  that you will see in the following slides!

21
An Example of Deconstructing the Test using the
7th Grade Periodic History Assessment

• Think about this On several standardized
  history tests, every question or answer related
  to the history of Ancient Islam that involved the
  name Muhammad, corresponded to a question or
  answer that included the word prophet.
• By looking at the tests, it becomes apparent that
  the concept that MUHAMMAD is viewed as a PROPHET
  by the followers of Islam, is important to the
  state of California and the test makers. (part of
  state standard 7.2.2)

22
In other words

• If you teach a student that knows nothing about
  the history of Islam, that when they see a
  question that includes the name Muhammad.., they
  should look for an answer that includes prophet.
  (keyed words)
• While this theory will not prove correct EVERY
  time, it does give students a leg up on answering
  questions that they may be unsure about.
• This is a better analytic strategy than simply
  choosing/guessing C, if they do not know the
  answer.

23
When in doubt choose C?...Not so fast..

• Statistically speaking choosing the letter C is
  not a better choice to use in standardized tests
  if one does not know the answer.
• This a MYTH! While there may have been a time in
  testing history that this was true, it is not
  true today.
• Modern metrical test design would never allow for
  and does not allow for such a repetition in an
  answering pattern.
• However, it is possible that at least any one
  test might be weighted towards certain answers
  A,B,C or D
• Sadly, it allows students to take the easy way
  out and not think or read the questions and can,
  in fact, hurt their overall scores.

24
Still dont believe deconstruction works.

• This question is straight from a 7th grade
  periodic history assessment. ----
• According to Islam, Muhammad was
• A the first to understand the meaning of the
  Christian Bible
• B an angel sent to earth by God to help the
  poor
• C an Islamic scholar who worked to preserve
  Greek Learning
• D the prophet through whom God revealed his
  final message

25
Lets try another a different way

• When you see a question involving Ancient African
  civilizations of Ghana or Mali, you should think
  about salt and gold trading.
• Ghana/Mali salt/gold trade ---(keyed words)
• Now answer this question using the keyed word
  strategy.
• This is an actual question from a recent periodic
  assessment.

26
Use the keyed word strategy.

• The economy of Ghana was dependent upon the trade
  of .
• A grain and salt
• B salt and gold
• C gold and oil
• D grain and slaves
• And the answer is B salt and gold
• Yes, it is a real question from a recent history
  periodic assessment.

27
More keyed word answers.

• On several recent periodic assessments and by
  looking at released questions if can be seen
  that
• Many 7th grade science questions that have to do
  with LIGHT are keyed to a response that includes
  the answer REFRACTION. Light Refraction
• Many 8th grade history tests that include a
  question about CHANGES to the CONSTITUTION are
  keyed to a response that has to do with the
  answer AMMENDMENTS. Constitutional changes
  amendments
• 7th grade questions about the development of
  CALENDARS are keyed to Mesoamerican cultures
  (AZTEC, MAYA, etc.) Calendar Maya or Aztec
• Get kids to understand KEYED WORDS!!!

28
To illustrate this deconstruction further lets
suppose that

• perhaps on a test, the questions that most
• students did poorly on were analyzed and it was
• found that those questions had
• More difficult vocabulary words (more use of
  academic words)
• These words contained three or more syllables
• These questions/answers contained longer sentence
  structures (maybe six or more words)
• These may be examples, but each may lead to an
  understanding of why some questions appear more
  difficult, even though we teachers hammer certain
  key concepts, only to have students get the
  answers wrong.

29
Visual Prompts in Testing

• Look at the WORLD MAPS above. Compare them.
• Think about how a struggling learner might see
  these two maps differently.
• Can you tell what the MAJOR difference is in the
  maps?

30
Color or Shade Inversion

• Many of the tests include maps, political
  cartoons, or diagrams that invert shades or
  colors based on differing questions.
• This type of test methodology is not an accident
  Test makers use this as a metric when designing
  the tests.
• These very subtle test design changes can cause
  major problems for the visually impaired, special
  education students and the wider test taking
  population as a whole.
• These shade inversions are actually studied by
  test makers and become part of the metrical
  design of future tests.

31
Brainstorming Deconstruction

• We need to look at every element of the test and
  find out why they are having difficulty with the
  questions.
• No detail is to be considered a waste of time.
• We need to be open to even the most minute
  trends, patterns, and themes.
• Do not assume the test taker will know certain
  basic words and phrases.
• Just because they can say it verbally does not
  mean they can read or write it!
• Again, NOTHING in the test design process is an
  accident. If we find even one new strategy per
  test by breaking down each assessment, we can
  improve overall statistical results for the whole
  campus.
• Assume nothing!!!!!!

• Lets look at
• Syllable counts
• Sentence length
• Word use frequencies
• Picture/chart/drawing use
• Color/shade inversion
• Use of italicization
• Use of boldfacing certain words or terms
• Use of underlining
• Number of questions per test
• Number of questions per page
• Location/placement of certain questions on a test
• ..anything that MIGHT help even if it turns out
  to be false.

32
But what about my subject matter?

• Can this work for Math Science English also?
• From what I have seen, it will work with science.
• I am not as familiar with the English and Math
  assessments, but while the strategy of keyed
  words might not apply to Math assessments, for
  instance, other deconstructive strategies might
  be found by the Math department to illustrate
  themes and trends in their own tests.
• The same goes for English!
• We need to tap our own department experts to find
  pre-assessment strategies that work.
• Remember, the Math, Science and English tests are
  no different than the History test from the
  design perspective.
• Psychometricians include certain types of design
  considerations, whether consciously or not, that
  can be exploited for our purposes in every test.

33
Vocabulary Review

• If students cannot read and comprehend the
  questions, then they cannot choose correct
  answers.
• Focus on Brick and Mortar words those more
  basic mortar words that connect academic words
  bricks. (Kate Kinsella)
• Need to focus on non-academic words and phrases
  that affect how students read questions -
  (occupied, scholar, through whom, preserve, etc.)
  These type of words actually trip up students
  based on surveys of my classes.

34
Non-Academic Vocabulary History words we should
never assume students already know

• alliances
• geographic (location)
• initial refusal
• unrelated
• related
• believe that
• inspiration
• faith(s)
• believed
• direct descendant(s) of,
• successful

• scholar
• preserve
• through whom
• monotheistic
• complex
• accurate
• developed
• economic (growth)
• spreading
• increasing
• defeat

35
Distracters defined.

• n.   One of the incorrect answers presented as a
  choice in a multiple-choice test.
• Some distracters are obviously wrong, while many
  are possibly correct or half-right.
• A test takers challenge is to weed out the BEST
  Possible Answer from the distracters.
• Our goal implement strategies and reinforce
  them in the classroom that teach students how to
  tackle these distracters.
• We need better strategies for dealing with
  distracters when test prepping students.

36
List Questions and Distracters

• Most questions involving lists of three or more
  concepts will use a negative (not, except,
  etc.) when written on a test.
• Example
• Which of the following is not considered to be
  one of the five pillars of Islam
• A - Pray five times a day
• B - Maintaining a special religious diet
• C - Pilgrimage to Mecca
• D - Give alms to the poor
• By previewing these types of questions and using
  the distracter format when teaching, we can give
  students an added boost in their thinking
  process.
• But this needs to become a conscious part of the
  teaching process!

37
Trick questions -

• Students should be aware of questions using
  negative words/terms such as
• which isexcept which is not
• all are....except except for
• none of the above all of the above
• --------------------------------------------------
  ------
• On average about a third of questions on any
  standardized tests use these except type
  questions.
• New strategies?

38
We have committees for

• Technology
• Discipline
• School site council
• Safe school issues
• Shared decision making
• Pearson/other learning curriculums.etc.
• But we DO NOT yet have an INSTRUCTIONAL committee
  to look at tests and test data
• You would think that this would have been
  implemented previously.. A no brainer!?

39
Instructional Committee?!

• Proposal a campus wide committee of one
  science, math, history, English, elective and
  physical education, and special education teacher
  along with one administrator should be formed to
  review all of the tests from all disciplines.
  Total of eight people maximum.
• And/or we can use the departmental approach and
  have each look at their own tests !
• Goal - This committees sole purpose should be to
  deconstruct the tests of each content subject
  area and find major themes, patterns and trends
  in each test that could be taught to the students
  before they take the assessments.
• Mission This committee should report back all
  findings to the staff during team, department and
  professional developments.

40
Instructional Committee Processes

• The entire group would look at each test
  (example all eight review the history test) and
  each person, regardless of whether they know
  anything about that subject area, will give their
  input about how they view the test (fresh eyes
  possibly looking at unfamiliar material).
• Example A science teacher might view the history
  test differently and point out something that a
  history teacher might have taken for granted and
  overlooked.
• Remember this is primarily about deconstructing
  the tests and not looking at only test data after
  the fact.

41
Instructional Committee Results

• After this process has been completed, each
  person from their specific disciplines will write
  a brief summary (bullet points) of their findings
  that the committee reviewed for their content
  area.
• The information will be given out to all faculty
  to use in their classrooms during everyday
  instruction and also during review periods for
  the different tests.

42
Warning!.......

• These strategies that result from the committee
  will NOT alone solve the testing issues we face.
• We must continue to use effective teaching
  methods, regular reviews and re-teaching of key
  concepts throughout the year.
• This process of deconstructing the tests is to be
  used as a support to regular instruction and
  should be used in conjunction with all post-test
  statistics.

43
Implementation.What do we do next?

• Use a three-fold strategy
• Deconstruct the tests to find vocabulary and
  terms that need to be re-taught to our students.
  Create subject area vocabulary lists to use as
  study sheets.
• Deconstruct the test design to find any themes,
  patterns or rhythms to the tests. Create lists of
  keyed words.
• Form and implement an Instructional Committee to
  evaluate the tests from the outside and
  deconstruct from the inside.

44
7th History Formative Assessment

• The following items have been evaluated based on
  their relevance and importance to this
  assessment.
• Generic breakdown of actual questions Actual
  numbers indexed to actual STAR test percentages
  in May.
• Thematic breakdown based on state standards -
  (contributions, cause/effect, core beliefs, etc.)
• Keyed Language words that should be taught
  together and used to link one to the other

45
(No Transcript)
46
Rome Questions - Breakdown
47
Islam Questions Breakdown
48
But I need proof this works?!...

• On the first 7th grade periodic assessment given
  in December 08, the questions that were
  evaluated in which I used these strategies showed
  a 92 or above score. Only one class had a
  slightly lower percentage in relation to one
  question and it was still at a respectable 88.
• Other History teachers also found a higher
  success rate using these strategies.
• Even special education students showed an
  increase in correct answers on those questions
  using the keyed word strategies.
• Dont believe it You are welcome to look at the
  my test results and see for yourself!

49
The future.?

• And now we can choose to take this further or
  not.
• The choices we make shortly could affect the
  outcome of our school funding.
• I do NOT profess to have all the answers or to
  claim that my theories and strategies will alone
  solve the problem. These ideas are but a
  suggested part of the test-prep whole.
• Thanks for your time!