Disproportionality and Best practices in Unbiased Assessment - PowerPoint PPT Presentation


PPT – Disproportionality and Best practices in Unbiased Assessment PowerPoint presentation | free to download - id: 1557c5-MTE5N


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Disproportionality and Best practices in Unbiased Assessment


Inaccessibility of the general curriculum to some ethnic minorities? ... Assess home, classroom teacher/student interaction; home/school interaction ... – PowerPoint PPT presentation

Number of Views:88
Avg rating:3.0/5.0


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Disproportionality and Best practices in Unbiased Assessment

Dis-proportionality and Best practices in
Un-biased Assessment
  • Steve Hirsch, PhD, NCSP
  • WSU EWU School Psychology program coordinator
  • East Valley, School Psychologist
  • smhirsch_at_wsu.edu

Best Realistic Practice in Non-Discriminatory
  • Completely unbiased assessment is an illusion.
  • Non-discriminatory assessment is not a search
    for an unbiased test but rather a process that
    ensures every individual, not just those who are
    deficit in some way, is evaluated in the least
    discriminatory manner possible. Ortiz, 2004

Best Realistic Practice in Non-Discriminatory
  • What is needed is a comprehensive, systematic
    framework comprising a broad range of methods and
  • But just as I am ready to dismiss standardized
    testing as invalid, I will present an intriguing
    model that is logical and allows for the use of
    standardized tests of cognitive functioning

Workshop Itinerary
  • The concept of dis-proportionality
  • The etiology of the concept
  • Cultural and Linguistic competency
  • Inadequacies of standardized tests
  • Value of formative assessment
  • Assessment of the Learning Ecology-FAAB
  • Our traditional attempts to reduce bias
  • The matrix of cultural/linguistic factor loading
    in cognitive assessment

  • Dis-proportionality only exists in the soft
    handicapping categories

(hard) Deafness Blindness Orthopedic
impairment Severe mental retardation
(soft) Specific learning disability Mental
retardation Emotional disturbance
In Washington
of pop. in SpEd
Child Count
2.7 4.0 16.4
Am Ind /AN 8.7 4.7
6 Asian/PI 5.7
7.7 14.8 Black 14.6
14.6 10.6
Hispanic 65.8 68.9
10.7 White
(No Transcript)
(No Transcript)
State Totals
Washington State
Summary Adequate Yearly
Progress 2007-08
Class of 2008 Washington State
Dropout Rates

Doubt is uncomfortable, certainty is ridiculous.

-Voltaire (François-Marie Arouet)
Where are we going wrong? From whence comes
  • Assessment/identification process?
  • Inaccessibility of the general curriculum to some
    ethnic minorities?
  • Is it a result of pre-conceived stereotypes?
  • Could it be that some populations are actually
    more prone to the deficits in the psychological
    processes that define a learning disability (i.e.
    memory, attention, processing speed, language,

Debate How is dis-proportionality in the
category SLD occurring?
  • Given
  • The most dis-proportionate populations have FS
    average scores significantly below whites on IQ
    testing (the tests are biased that way)
  • To be identified as LD you must have a
    significant discrepancy between IQ and
    achievement on standardized test
  • We all know that it is nearly impossible to
    qualify a child with borderline to low average IQ
  • How then are we qualifying a disproportionate
    number of blacks and Native Americans as LD?
  • Professional Judgment?

How did we get to this point? Could
dis-proportionality be a result of how we view
special education?
  • From WAC, special education represents
    brokenness of the nervous system. Keep em
    out of sped
  • To the ELL parent, special education may
    represent what it was in the previous culture.
    keep em out
  • To an advocacy group, special education often
    represents a label, stigma but protection in form
    of IEP keep em out unless they qualify
  • To the team, special education often represents
    additional help usually in a small group or
    individual setting. E.g. tutoring. put them in

Could dis-proportionality be a result of how we
view special education?
  • If special education is viewed as a way of
    helping a child using resources not available to
    the general education community, or is viewed as
    protection in form of IEP, then how could we not
    expect certain minority populations to be
  • They dont test academically as well on our
    standardized achievement tests and therefore are
    more likely to look as if they need extra help.

Best Realistic Practice in Non-Discriminatory
Assessmentala APA
  • Consider influence of language and culture on
  • Consider validity of methods and procedures
  • Make interpretations of data within context of a
    students linguistic cultural characteristics
  • When in doubt, hide behind vagueness

Needed Cultural Linguistic Competency
  • Cultural competence- knowledge base of, or direct
    experience (with values, attitude, beliefs and
    customs of particular culture) that will guide
    data collection analysis.
  • Doesnt come from a book or taking a trip or even
    looking out your kitchen window at another
    country. Look for professional development or
    experiences with community or advocates

Willingness To see and Accept help
Acculturation indicators
Perception of Disability
Needed Cultural Linguistic Competency
  • Linguistic competence Ability to communicate
    effectively in individuals native language (no
    interpreter)---i.e. multi-lingual
  • Typically not reality so next best thing
  • Possession of a knowledge base related to first
    and second language development and instructional
    methodology as it relates to ELL

And if you are going to use standardized
tests-skills you need to know
  • Adequacy of sampling norms
  • Full range of abilities being measured in
  • Linguistic demands and cultural loading
  • Reliability and validity of cultural-fair tests
  • Sources of potential bias

And if you are going to use standardized
tests-keep in mind that
  • Published tests of cognitive ability are often
    viewed as psychometrically non-biased due to
    sampling norm that mirrors census data.
  • Acculturation impacts appropriateness of tests
  • Decisions must be made on individual basis

Debate Is a particular cultural group ever
adequately represented in a normative sample?
  • If 3 of test sample represents a group that
    constitutes 3 of the population, does that mean
    that the test is appropriate for your particular
  • The 3 in test sample-to what extent are they
    representative of their cultural population?
  • To what extent is your student representative of
    their cultural/linguistic population?
  • Just because a culture is represented in a sample
    doesnt mean that the test is fair to the culture

Adding insult to injury
  • If you look at sampling norm data in test
    manuals, you typically find the following
    (TONI-3, 1997)
  • Carefully stratified across US geographic regions
    (northeast midwest south west-census matched)
  • Ethnicity categories- African American Hispanic
    Asian Native American.
  • While they are careful not to conclude that a
    mid-westerner is similar to a westerner, they
    assume that Chinese, Vietnamese, Korean, Hmong,
    Indonesian, Filippino, Samoan, Indian, Japanese
    and Hawaiian among hundreds of other cultures are
    all identical

Other important issues
  • Degree to which the test is linguistically loaded
  • Limited verbal load (WISC perceptual
  • Non-verbal Minimal verbal interaction (KABC)
  • Language-free (UNIT)
  • Degree to which a test, even if language-free, is
    culturally loaded
  • Bottom line we know very little how different
    bilingual populations perform on tests normed on
    children from monolingual single-culture

What assessment tools are available, and what
could be assessed Cognitive
  • Reduced Language Load (WISC-PO index)
  • Would you want the world to believe that your IQ
    is solely a function of non-verbal subtests? Not
  • Non-verbal assessments (KABC-II DAS)
  • If instructions frame problem-solving strategy
    but instructions are in English
  • Language-free assessment (UNIT)
  • But cognitive functioning is in large part, verbal

What assessment tools are available, and what
could be assessed Academic
  • Translated assessments e.g. Woodcock-Munoz
  • Problem is that bilingual children may never
    received formal instruction in their native
    language. Academic achievement must be viewed
  • Curriculum Based Measurement (ORF) when used with
    ELL students
  • Has been shown to have good reliability
  • Has been shown to have good validity
  • Has been shown to be a reliable progress monitor

Academic Assessment of ELL Life is good-its
almost time for Gonzaga basketball epiphany
  • Global norm-referenced tests invalid
  • Translated versions of tests-questionable
  • ORF susceptible to word-calling
  • What if we looked at more than the correct number
    of words read per minute during DIBELS? What if
    we focused on a composite of speed, accuracy and
    prosody? True Oral Reading Fluency. Will it
    correlate with overall reading or WASL reading?
    Do ELL students improve in prosody? Need

Academic Assessment of ELL fear of shoveling 10
ft of snow epiphany
  • We have all sorts of benchmarks for ORF but none
    reflect the ELL student. Are there unique
    learning trajectories in ORF for ELL students
    based on their English language proficiency? But
    how does acculturation degree factor into this?

What assessment tools are available, and what
could be assessed Instructional Environment
  • Do students have sufficient background to
    understand content of instructional materials?
  • Can students relate to content of instructional
    materials from cultural perspective?
  • Is curriculum being adapted to needs of bilingual

What assessment tools are available, and what
could be assessed Instructional Environment
  • A difficult task Evaluate teacher expectations
    of bilingual students and how it might impact
  • Do we have adequate benchmarks to assess the
    progress of ELL students against?
  • Interesting study for someone out there How are
    teachers expectations of a student influenced by
    students cultural background?

Do teachers have different expectations of
children of differing cultural backgrounds?
  • Based on your interactions with children of these
    cultural backgrounds, rank them in terms of the
    likelihood of this child returning homework daily
  • ___Russian
  • ___Hmong
  • ___Samoan
  • ___Vietnamese
  • ___Hispanic
  • ___Chinese
  • ___Native American

What assessment tools are available, and what
could be assessed Instructional Environment
  • The Real Danger Shared 40 yrs ago
  • Standardized tests label black children as less
    educable place black children in special
    classes perpetuate inferior education assign
    black children to lower education tracks than
    whites deny black children higher educational
    opportunities and destroy positive intellectual
    growth and development of black children.

Are standardized tests really that poor for ELL
  • Norms usually limited to small samples of
    minority children
  • Norms routinely exclude students with limited
    English proficiency
  • Test items tap info that minority children may
    not be familiar with
  • Tests typically dont allow examiners to probe
    and inquire further
  • Correct items on tests usually based on majority
    childrens responses
  • Standardized testing procedures assume that
    students have appropriate test-taking skills

What about translated tests?
  • Not really the answer
  • Sometimes vocabulary and concepts not easily
    translated (e.g. Yiddish word, mensch not
    easily translated to English)
  • Difference in dialects result in translation
  • Some concepts change meaning once translated

So if not standardized tests, what ?
  • Then we will look at Best Practices in
    non-discriminatory assessment- A comprehensive

  • For every complex problem there is an answer
    that is clear, simple, and wrong.
  • H. L. Mencken (1880-1956)

Comprehensive Framework
  • Evaluate the learning ecology
  • Evaluate language proficiency
  • Evaluate opportunity for learning
  • Evaluate educationally relevant cultural and
    linguistic factors
  • Develop, and revise hypotheses

Assessment of learning ecology
  • Exploration of extrinsic factors that might lead
    to difficulties
  • Functional Assessment of Academic Behavior (FAAB)
  • Interviews w/parent, teacher, child
  • Observations in multiple settings
  • Assess home, classroom teacher/student
    interaction home/school interaction

Assessment of opportunity for learning-specific
factors to be examined
  • Regularity of attendance
  • Experience with school environment
  • Language match (native vs. English)
  • Parents ability to support language instruction
  • Years of instruction in English

Assessment of opportunity for learning-specific
factors to be examined
  • Quality of ESL instruction or instruction in
    native language
  • Frequency of school change
  • School attitude expectations re. dual language
  • Socialization vs. isolation from peers

Assessment of educationally relevant cultural and
linguistic factors
  • Learning takes place everywhere, not just school.
    Many factors outside educational setting can
    affect the learning process.
  • Exposure to two or more cultures or languages
    during early childhood may create circumstances
    that cause child to have very different
    experiences than other children.

Evaluate and revise hypotheses
  • Only when enough confidence exists in the belief
    that there are no plausible external factors that
    could account for students learning difficulties
    is it time to consider internal factors.
  • There should also be evidence that efforts to
    reduce impact of external factors through
    intervention were unsuccessful.

So far
  • The above steps should all be part of a
    pre-referral process not part of the special
    education assessment

Reduce bias in traditional testing practice
  • Bias is due to non-representational sampling- key
    is acculturation and English proficiency. Even
    with native language tests, bias remains. Two
    traditional approaches to reduce bias

Traditional approaches to reduce bias
  • Administer tests in standardized way-attempt to
    evaluate results in non-discriminatory manner
  • Use of existing locally developed norms-more
    appropriate comparison group. Standardization
    then helps for comparison
  • Deviations from standardization would produce
    results that are unknown and unpredictable as
    does translated tests
  • Test selection based on properties relating to
    cultural loading

Traditional approaches to reduce bias
  • Modify testing process in a way that is less
  • Any modification will of course violate
    standardization and impact validity and
    interpretability of results
  • You also lose ability to use results for
    comparison purposes
  • We typically dont know impact of any
  • We could look at data qualitatively

Traditional approaches to reduce bias some
  • Bilingual administration
  • Use of extended-expanded instructions on sample
  • Mediation of concepts to ensure comprehension
    prior to test administration
  • Repetition of items to facilitate comprehension

Traditional approaches to reduce bias some
  • Extension or elimination of time limits
  • Acceptance of alternate responses e.g. non-verbal
  • Additional probing and querying of incorrect or
    partially correct answers

Utilize authentic assessment procedures
  • Whereas standardized norm-referenced tests are
    driven by questions and needs related to
    classification, diagnosis and legal eligibility,
    authentic assessment is geared more towards
    answering questions regarding instructional needs
    and interventions.

Evaluate data within context of learning ecology
  • Interpret and analyze in terms of acculturation
    and English language proficiency-cohort analysis
    through OSPI
  • Knowledge of factors that may have played a part
    in creating significant differences between the
    experience of the individual in terms of
    acculturation or language development- provides
    the least discriminatory framework with which to

Evaluate data within context of learning ecology
  • Care should be taken not to give too much
    importance to any single piece of information.
    There is a tendency to jump on any data that is
    inspirational in that it was unexpected.

Linking Assessment to Intervention
  • Assessment and evaluation are not interventions!
  • Problems do not disappear as a result of
  • Assessment is of little value without appropriate
    interventions and treatment options. Does the IQ
    really yield valuable info re. intervention?
  • The non-discriminatory assessment yields data
    ripe for intervention

Is this leading up to RTI? You betcha! (Palin,
  • The process is the same for the bilingual student
    as any student
  • Identify the at-risk student. Not all bilingual
    students are at-risk
  • Use diagnostic assessment (incl. FAAB) to
    determine appropriate interventions. Not all
    bilingual students need the same interventions
  • Monitor progress during intervention
  • Evaluate effectiveness of intervention and modify
    intervention if necessary

Drop out rates
Graduation rates
Proficiency on State assessments
Academic Achievement
Disproportionate representation in special ed
Assessment under RTI
  • CBMs are probably the most efficient and maybe
    the most valid tool for identifying the at-risk
    bilingual student
  • In reading- ORF(e.g. DIBELS) Aimsweb
  • In math- Aimsweb, MathAlert! Or MAP
  • In writing- 1min think/3 min write

Assessment under RTI
  • Diagnostic evaluation for bilingual students
    under RTI can include
  • FAAB for learning ecology
  • Reading QRI DRA BRI assess five essential
    reading components
  • Math KeyMath MAP
  • Written language in response to prompt assess
    seven essential elements

Essential Reading and Writing Components
  • Reading phonemic awareness phonological
    processing fluency vocabulary comprehension
  • Writing
  • Fluency (total words written)
  • Legibility (letter formation analysis)
  • Conventions (spelling/punctuation/capitalization)
  • Syntactic Maturity (varied sentence
  • Semantic Maturity (variety of words/vocabulary)
  • Content (organization, cohesion)
  • Writing Process( plans ahead, transitions)

Where does this all leave you today? In my opinion
  • No valid standardized test
  • Assessment a framework or process, not a test
  • Assess the learning or instructional environment
  • Knowledge about the specific culture
  • Knowledge about bilingual education
  • Typical learning progress rates of ELL students

  • Just when I thought my case was airtight and my
    message was clear

A model for the cognitive assessment of ELL and
culturally diverse students that makes sense
  • Subtests on the various cognitive functioning
    tests (e.g. WISC IV WJIII) can be rated on their
    sensitivity to either linguistic or cultural
  • see the NASP publication Ortiz (2004)
    Comprehensive Assessment of Culturally and
    Linguistically Diverse Students

Comprehensive Assessment of Culturally and
Linguistically Diverse Students
  • Examining the pattern of scores on the subtests,
    taking into account this sensitivity, might allow
    us to discriminate between
  • cognitive difficulties resulting from a lack of
    language/cultural experience or something else
    (i.e. disability)

Matrix of Cultural Loading and Linguistic
Demand(WISC IV)
Dependency on linguistic competency
Low Medium High

Dependency on cultural competency
High Medium Low
And is all this realistic?
  • Just my preaching!
  • Make this a professional goal for the next year
  • WSASP will provide professional dev. opportunity
  • Familiarize yourself with the cultures of your
    community-attend cultural activities church etc.
  • What is realistic is to learn more about the FAAB

It is easier to build strong children than to
repair broken men.
Frederick Douglass
About PowerShow.com