The Health Literacy Component HLC

1
The Health Literacy Component (HLC) of the 2003
NAAL
Sheida White Project Officer National Assessment
of Adult Literacy (NAAL)
2
Topics covered by this presentation

• Background, key features, and major goals
• Performance of literacy tasks
• Skills required to perform literacy tasks
• Assessment design and administration
• Data analysis and reporting

3

• Background, Key Features, and Major Goals

4
NAAL background

• In 1985 and 1992, the National Center for
  Education Statistics (NCES) conducted adult
  literacy assessments that focused on real-world
  tasks
• The 2003 National Assessment of Adult Literacy
  (NAAL) builds on and expands previous NCES adult
  literacy assessments
• NAAL is the first assessment of the nations
  progress in adult literacy since the 1992
  National Adult Literacy Survey (NALS)

5
Key features of NAAL

• Is based on in-person interviews of approximately
  18,000 Americans age 16 and older
• Provides data for the United States as a whole,
  individual states choosing to participate in a
  state-level assessment, and the nations prison
  population
• Emphasizes the use of printed everyday materials
  (such as bills, prescriptions, newspapers) needed
  to function adequately in ones environment
• Provides data on background characteristics,
  performance of everyday literacy tasks, and
  skills underlying task performance

6
Major goals

• Describe the status of adult literacy
• Report on trends between 1992 and 2003
• Identify relationships between literacy and
  selected characteristics of adults
• Provide new information about skills underlying
  adult literacy, including basic reading skills of
  the least-literate adults
• Facilitate use of NAAL data by diverse audiences,
  including policymakers, researchers, and educators

7

• Performance of Literacy Tasks

8
NAALs task-based literacy definition
Literacy is the ability to use printed and
written infor-mation prose, document, and
quantitative to function in society, to achieve
ones goals, and to develop ones knowledge and
potential
S. White 8-04
9
Definitions of three key literacy areas
10
Measurement of prose, document, and quantitative
literacy

• NAALs prose literacy, document literacy, and
  quantitative literacy scale scores are comparable
  to those of the 1992 NALS
• Scores are based on completion of items that are
  all
• Classified as prose, document, or quantitative
• Open-ended to represent everyday tasks
• Placed before stimulus materials to set a purpose
  for reading
• Most items require
• Short written responses
• Searching the text for specific information

11
Topics of NAAL stimulus materials
12
Sample prose item used in 1992

S. White 8-04
13
Sample document item used in 1992
S. White 8-04
14
Sample quantitative item used in 1992
15
A new health literacy score based on
health-related items

• NAAL includes 28 health-related items (out of 153
  items)
• All participants receive some health-related
  items
• All health-related items are also classified as
  prose, document, or quantitative items
• The health-related items
• Are included with other items when calculating
  the prose, document, and quantitative scale
  scores
• Are used to calculate a separate health literacy
  score, based solely on health-related items

16
Development and focus of the NAAL Health
Literacy Component

• NAALs Health Literacy Component was developed in
  response to a request by the U.S. Department of
  Health and Human Services (HHS)
• NAAL focuses on one key aspect of health
  literacythe ability to use printed and written
  materials to accomplish a broad range of
  health-related tasks
• NAALs health-related tasks include
• Clinical tasks, having to do with the health care
  itself
• Preventive tasks, having to do with healthy
  habits and prevention of illness
• Navigation tasks, having to do with bureaucratic
  demands to get to the health care

17

• Skills Required to Perform Literacy Tasks

18
Purpose of the Adult Literacy Supplemental
Assessment (ALSA)

• The 1992 NALS provided little information about
  the literacy abilities of the least-literate
  adults, who were not able to complete the
  assessment
• The purpose of the new Adult Literacy
  Supplemental Assessment (ALSA) is to get a
  clearer picture of the skills of the
  least-literate adults
• ALSA is intended only for adults who would not be
  capable of meaningful participation in the main
  NAAL assessment, because they lack the skills to
  perform even the easiest tasks with a high
  degree of success
• These adults take ALSA instead of the main NAAL

19
Selection of ALSA participants

• All NAAL participants take seven easy core
  screening items
• Interviewers score participants answers using
  the scoring guides provided on their computers
• Computer selects ALSA participants using an
  empirically derived algorithm that predicts very
  low performance
• The algorithm predicts which participants would
  get a total of five items or fewer correct
  (including the core items)
• Five items, spread across the three NAAL scales,
  would not be sufficient to allow accurate
  estimates of performance on any of the scales

20
Important features of ALSA

• ALSA instructions and responses are given orally
  and can be in either English or Spanish
• ALSA materials are
• Contextualized (with logos, pictures, etc.)
• Familiar (as measured by background questions)
• Tangible (e.g., food boxes, medicine bottles,
  utility bills)
• ALSA tasks, although simple and contextualized,
  still require participants to read at least parts
  of the words

21
The need for additional data about skill deficits

• In addition to the 6 of adults who lacked the
  skills to complete the 1992 NALS, an additional
  15 of adults were able to perform only the
  easiest NALS tasks (classified as Level 1
  tasks)
• For adults with low scores, more information is
  needed about the nature of the skill deficits
• Lack of basic (word-level) reading skills?
• Lack of fluency in basic skills?
• Lack of higher level literacy skills?

22
Range of skill deficits and their potential
implications
23
New data on the skills of all adults

• The Fluency Addition to NAAL (FAN) provides new
  data on the skills of all adults
• After completing either the main NAAL or ALSA,
  all participants take FAN, which
• Consists of timed oral reading tasks
• Describes the oral reading skills of adults
  (e.g., words read correctly per minute)
• Provides a measure of basic (word-level) reading
  skills
• FAN scores can be compared with main NAAL scores
  at various performance levels, ALSA scores, and
  health literacy scores

24
FAN oral reading materials and associated
measures

• Pseudoword lists, consisting of possible but
  nonoccurring English forms (e.g., wike),
  provide a measure of adults ability to decode
  (or identify the sounds of) words with which they
  are not familiar
• Word lists, consisting of English words arranged
  in increasing order of difficulty, provide a
  measure of adults ability to recognize familiar
  words (often referred to as sight words) as
  well as to decode
• Text passages, consisting of 150200 words each,
  provide a measure of adults ability to read
  words in connected texts

25
Initial measures and scoring challenges for FAN

• Initial FAN target measures are
• Total words read aloudwhether correctly or
  notper unit of time
• Words read correctly per unit of time
• Words read correctly as a percentage of total
  words read
• Challenges in scoring FAN data include ensuring
• That correctness can be measured reliably
• That speakers of nonstandard varieties of English
  are not unfairly penalized
• Unit of time 60 seconds for text passages or 20
  seconds for lists

26
Key similarities and differences between FAN and
ALSA

• Similarities include the following
• Both measure basic reading skills
• Both have oral instructions in either English or
  Spanish
• Both require reading words and connected text
• Differences include the following
• Only ALSA permits the use of compensatory
  strategies, while FAN does not provide any
  nonlinguistic clues (such as pictures)
• ALSA connected texts are fewer, easier, and
  shorter (1?2 sentences each in ALSA, compared
  with 150?200 words each in FAN)

27
The functionality of word-level skills and
higher level literacy skills

• NAAL defines literacy as the ability to use
  printed and written information
• As defined by NAAL, all literacy is functional
  therefore, NAAL does not differentiate between
  literacy and functional literacy
• Certain skills, including word-level reading
  skills, are needed to successfully perform NAAL
  tasks
• The ability to read words is a functional skill,
  but not the only one
• In addition to word-level skills, NAAL recognizes
  six types of literacy skills

28
The six types of literacy skills
NAAL literacy skills
Language comprehension skills
Text search skills
Inferential skills
Understanding the structure and meaning of
sentences
Drawing appropriate text-based inferences
Searching textefficiently
Computation performance skills
Computation identification skills
Application skills
Identifying the calculations required to solve
quantitative problems
Performing any required calculations
Using newly searched, inferred, or computed
information to accomplish a variety of goals
29
Analysis of main NAAL data in terms of underlying
skills

• Recently developed multidimensional analysis
  methods will be applied to 2003 main NAAL data in
  order to
• Characterize the proficiency of adults in terms
  of the NAAL literacy skills
• Characterize effects of task demands and written
  materials on the difficulty of literacy tasks

30

• Assessment Design and Administration

31
Selection of the national sample for NAAL
32
Key features of the NAAL sample

• Adults age 16 and over living primarily in
  households
• Nationally representative sample of 10,006
  adults, augmented by
• Samples from the six states that participated at
  the state level (an additional 7,166 adults)
• Prison sample of inmates (about 1,000 more
  adults)
• Incentive payment of 30 to increase
  representativeness of the sample and response
  rate
• Oversampling of Blacks and Hispanics

33
Matrix sampling

• The main NAAL has a total of 153 items (including
  the 7 core screening items that are given to all
  participants and used to select ALSA
  participants)
• Matrix samplingadministering only a portion of
  the non-core items to each respondentensures
  broad coverage while limiting respondent burden
• Each respondent takes 1 of 26 booklets
• Each booklet includes 3 of 13 blocks of items
  (for about 40 items per booklet, including the
  core items)
• Each block is spiraled in the 26 booklets (i.e.,
  paired with every other block)

34
Allowing comparisons with the 1992 NALS while
introducing new features
1992 assessment
2003 assessment
Six of the blocks from 1992 are used again in
2003. Seven new (2003) blocks replace all six of
the blocks from 1985 plus one of the blocks from
1992.
13 blocks total
13 blocks total
S. White 8-04
35
Allowing comparisons with the 1992 NALS while
introducing new features (continued)

• The seven blocks that were newly created for 2003
• Are similar to the 1992 blocks in terms of the
  distribution of skills required by tasks
• Have approximately the same average task
  difficulty as the 1992 blocks
• Are spiraled with the 1992 blocks in the 26
  booklets
• Item response theory will be used to link the
  1992 and 2003 scales using the tasks common to
  both assessments

36
New performance levels for 2003

• In 1992, five performance levels were developed
  by grouping together tasks of similar difficulty
• The 1992 levels proved to have some limitations
• Scores that are the cut points between levels
  vary depending on the measure of task difficulty
  used
• No distinctions are made within the lowest level
  of performance (Level 1)
• For 2003, new performance levels are being
  developed for the main NAAL by the National
  Academy of Sciences
• Goals of the new levels include increased clarity
  and better differentiation among adults at the
  lower end of the scale

37
NAAL total average assessment time

• 
  Main NAAL ALSA
• Background questionnaire 28 min 28 min
• Core screening items 8 min 8 min
• Main NAAL or ALSA assessment 39 min 25 min
• FAN 14 min 12 min
• Total 89 min 73 min
• Respondents may take as long as they need to
  perform main NAAL or ALSA tasks, but are
  encouraged to move to a new task when stuck or
  frustrated

38
Accommodations for adults with special needs

• The assessment is conducted in the participants
  home
• The assessment is administered one on one
• All participants receive additional time to
  complete the main NAAL or ALSA if they need it
• The background questionnaire is administered
  orally in either English or Spanish
• Instructions for the core screening items and for
  ALSA are given orally in either English or
  Spanish
• Participants with a native language other than
  English or Spanish may take the core screening
  items even if they cannot complete the background
  questionnaire

39
Expanded NAAL background questionnaire

• For 2003, the background questionnaire was
  expanded to better reflect
• Economic and technological developments in the
  past decade
• The background of low-performing adults
• Health-related activities of adults
• Literacy-related activities of parents and their
  children

40
Characteristics of the NAAL scoring
rubrics Statistical considerations

• Responses to each item are analyzed for
• Frequency with which a particular response is
  given
• Interrater reliability in scoring responses
• P-value (percentage of the population responding
  correctly)
• Biserial (and polyserial) correlation between
  each response (and partially correct response)
  and the respondents overall score
• Range findings for delineation of the range of
  responses received
• Differential item functioning (DIF) in favor of
  certain groups

41
Development of the NAAL scoring rubricsContent
considerations

• Seek evidence that adults can indeed use printed
  materials
• Allow for partially correct responses if the
  information provided is still useful in
  accomplishing the task
• Score most responses as either correct or
  incorrect, since often a partially completed task
  is not an accomplished task
• Allow for writing errors as long as the overall
  meaning of the response is correct
• Distinguish responses providing specific
  information from those providing generalizations

42

• Data Analysis and Reporting

43
-
New software for data analysis

• AM software has been developed by the American
  Institutes for Research (AIR), with NCES funding
• The new software will reduce the amount of time
  required to analyze data by several months
• The software will allow the technical report and
  the Data File Users Manual to be produced on a
  more timely schedule than in 1992
• The software will also make it easier for states
  and researchers to do their own data analyses to
  supplement the information in the state and
  national reports

44
Average scores and distributions

• Average scale scores and score distributions will
  be
• provided
• For the population as a whole
• For population groups
• But not for individuals

45
Inappropriateness of individual scores

• No respondent takes all of the assessment
• It would not be appropriate to calculate
  individual scores based on part of the assessment
  because
• The individual scores would be unreliable (a
  reliable or consistent measure requires many
  items)
• The individual scores would not be a valid
  representa-tion of the domain of adult literacy
  (valid representation requires many types of
  tasks, genres, etc.)

46
Trend analysis

• Results from the 1992 NALS and the 2003 main NAAL
  will be compared in terms of
• Average scale scores for prose, document, and
  quantitative literacy
• Performance levels set by the National Academy of
  Sciences
• P-values (the percentage of the population giving
  the correct answer to each item)

47
New release of items and p-values

• About 90 assessment items used in 1992 assessment
  will soon be released
• All released items will be available on the NAAL
  website (http//nces.ed.gov/naal)
• Each items p-value will be published
• This will be the first publication of p-values
  for 1992 assessment items
• P-values will be published for all items used in
  1992even items that are not released

48
Test Questions Search Tool on the NCES website
(http//nces.ed.gov/naal)Sample screen 1

49
Test Questions Search Tool on the NCES website
(http//nces.ed.gov/naal)Sample screen 2

50
New types of results in 2003

• Health literacy scale scores measuring the
  performance of American adults on health-related
  tasks
• Skill-based analyses inferring the literacy
  skills associated with the ability to perform
  main NAAL tasks
• ALSA scores providing information on the literacy
  skills and deficits of Americas least-literate
  adults
• FAN oral reading and basic skills data providing
  information about the skills of all adults

51
NAAL mailing list

• If you would like to receive NAAL information
  tailored to your interests, simply fill out a
  short online form at http//www.edpubs.org/list/NA
  AL
• On the form, you can indicate up to 16 areas of
  interest
• Adult basic education
• Adult secondary education
• Citizenship/civics education
• College prep
• Computer literacy
• Correctional education
• Early childhood/childrens literacy
• ESL/ESOL (English as a second language)

• Family literacy
• Health literacy
• Homeless education
• Learning disabilities
• Mathematical literacy
• Native language literacy (i.e., literacy in
  Spanish)
• Policy and legislation
• Workplace literacy

52
National Center for Education Statistics
Sheida WhiteNAAL Project Officer(202)
502-7473Sheida.White_at_ed.gov Andrew
KolstadSenior Technical Adviser(202)
502-7374Andrew.Kolstad_at_ed.gov Websitehttp//nc
es.ed.gov/naal