AUTHENTIC%20ASSESSMENT%20AND%20PROGRAM%20EVALUATION%20IN%20EARLY%20CHILDHOOD - PowerPoint PPT Presentation

About This Presentation
Title:

AUTHENTIC%20ASSESSMENT%20AND%20PROGRAM%20EVALUATION%20IN%20EARLY%20CHILDHOOD

Description:

0.Is not able [can't; too young; physical limits] 1.Never/almost never when ... Is restless and 'fidgety' 0 1 2 3. Withdraws from the company. of others 0 1 2 3 ... – PowerPoint PPT presentation

Number of Views:290
Avg rating:3.0/5.0
Slides: 93
Provided by: karashairr
Learn more at: https://nectac.org
Category:

less

Transcript and Presenter's Notes

Title: AUTHENTIC%20ASSESSMENT%20AND%20PROGRAM%20EVALUATION%20IN%20EARLY%20CHILDHOOD


1
(No Transcript)
2
(No Transcript)
3
(No Transcript)
4
Scaling Progress in Early Childhood Settings
(SPECS)STEPHEN J. BAGNATO, Ed.D.,
NCSPProfessor of Pediatrics PsychologyDirector
, Early Childhood PartnershipsDirector, SPECS
(PAPREKA/PEIOS)Childrens Hospital of
Pittsburgh/UCLID CenterUniversity of Pittsburgh
School of Medicinesteve.bagnato_at_chp.eduwww.uclid
.org
5
SPECS Program Evaluation Research Team Leaders
PAPREKA and PEIOSStephen J. Bagnato, Ed.D.,
NCSP, Director Candace Hawthorne, Ph.D., OTR/L,
CoordinatorIlene Greenstone, MA, CoordinatorPip
Campbell, Ph.D., OTR/L, CoordinatorAssisted by
Western and Eastern PA Research Teams
6
What is the Authentic Assessment Alternative to
Conventional Testing in Early Childhood?
7
(No Transcript)
8
Inauthentic Measurement in Early Childhood
  • Much of developmental psychology early
    childhood assessment as it now exists is the
    science of the strange behavior of children with
    strange adults in strange settings for the
    briefest possible periods of time.
    (Bronfenbrenner, 1979, p. 19)

9
(No Transcript)
10
(No Transcript)
11
Authentic Assessment in Early Childhood
  • Natural observations of ongoing child behavior in
    everyday settings and routines vs. contrived
    arrangements
  • Reliance on informed caregivers (teachers,
    parents, team) to collect convergent,
    multi-source data across settings
  • Curriculum-based measures linked to program
    goals, content, standards, expected outcomes
  • Universal design equitable assessment content
    and methods
  • Intra-individual child progress supplemented by
    inter-individual normative comparisons
  • NAEYC/DEC/HS PA DAP Assessment Standards
    Practices

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Are There Professional and Pennsylvania Standards
for Authentic Assessment in Early Childhood/Early
Intervention?
16
(No Transcript)
17
(No Transcript)
18
Selected Professional Standards for Early
Childhood Assessment (DEC, 2004 NAEYC, 1997,
HS, 2000)
  • Reliance on developmental observations-ongoing
    observational assessments overtime
  • Performance on authentic, not contrived,
    activities
  • Integration of assessment and curriculum
  • Child progress on past performances as the
    reference, not group norms
  • Choose materials that accommodate the childs
    special functional needs
  • Use only measures that have high treatment
    validity
  • Rely on curriculum-based measures as the
    foundation or mutual language for team
    assessments
  • Defer a diagnosis until evaluation of a childs
    response to a tailored set of interventions
  • Use scales with sufficient item density to detect
    even small increments of progress

19
6 Best Practice Criteria for Authentic
Assessment in Pennsylvanias ECE Programs
  • Purpose Assess for program planning not
    diagnosis or exclusion eliminate readiness
    testing practices
  • Method No tabletop testing Deemphasize
    scores observe and assess functional skills that
    link to the curriculum standards
  • Context Observe and record evidence of natural,
    ongoing child development and behavior in
    typical, everyday routines not contrived settings
  • Process Rely on teachers/caregivers to observe
    and record child progress 2-3 times each year
  • Standards-based Align all assessments and their
    item content with ELS and curricula link
    assessment with expected outcomes
  • Parent Partnerships Enable parents to have
    central role in providing observational data on
    progress

20
Measures for PEIOS/PAPREKA Research Balance of
Attributes
  • Tension to balance research rigor with utility
    Choice of measures based on following elements
  • Simplicity
  • Authenticity
  • Utility
  • Evidence-base
  • Standards-referenced
  • Functional content
  • Sensitivity

21
(No Transcript)
22
The Pennsylvania Early Intervention Outcomes
Study(PEIOS)Documenting the Benefits of Early
Intervention Supports in PA to Fulfill State and
Federal Mandates
23
What are the missions, research questions and
authentic measurement design for PEIOS?
24
PEIOS Fast Facts
  • Aim Document early intervention outcomes for
    state federal mandates
  • County agencies and MAWAs in 6 PA Regions mapped
    to PQP
  • Random selection
  • Collect data on entry-level functioning compared
    to performance at followup
  • Use both/either an existing measure and a common
    functional measure across programs ABAS II
  • File reviews to code program and service
    intensity
  • Multiple research strategies to analyze
    interrelationships among program intensity and
    child/family outcomes
  • Classify outcomes by OSEP/ECO categories

25
PEIOS Outcome and Research Measures
  • Child Measure Caregiver
  • Adaptive Behavior Assessment System II (ABAS
    Harrison Oakland, 2004)
  • Child Measure Caregiver
  • Program-Identified Measure (BDI DC COR)
  • Program Measure PEIOS Team
  • Program Specs (Bagnato, 2005)
  • Program Measure PEIOS Team
  • Developmental Specs (Bagnato, 2005)

26
(No Transcript)
27
Adaptive Behavior Assessment System II (ABAS,
2003)
  • Multi-dimensional observational judgment-based
    rating scale (3-point) of functional competencies
  • Ages 0-89years
  • Early childhood forms Parent Teacher/Provider
  • Nationally standardized Ages 0-52100 all forms
    and all ages5270
  • Norm-referenced scores General Adaptive
    Composite (100,15) Subskills (10,3)
  • Excellent technical research base disability
    studies Aligns with DSM IV AAMR categories
  • Psychological Corporation

28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
ABAS Rating Format
  • 0..Is not able cant too young physical
    limits
  • 1..Never/almost never when needed prompts
  • 2..Sometimes when needed with/out help
  • 3..Always/almost always when needed before
  • G..Check if you guessed

33
ABAS Domains Parent Form (241) Teacher (216)
  • Communication
  • Community Use
  • School Living/Home Living
  • Functional Pre-Academics
  • Health Safety
  • Leisure
  • Self-Care
  • Self-Direction
  • Social
  • Motor

34
ABAS Disability Research Clinical and Matched
Control Samples
  • Mental Retardation
  • Developmental Delay
  • Biological Risk Factors (e.g., prematurity
    drugs)
  • Motor and Physical Impairments
  • Language Disorders
  • Autism and PDD
  • Learning Disability
  • ADHD
  • Alzheimers Disease
  • Neuropsychological Disorders
  • Behavior and Emotional Disorders
  • Deaf and Hard of Hearing

35
PEIOS Program Evaluation Research Measurement
Model
  • January 2006 Identify and train PEIOS program
    evaluation liaisons
  • January-February 2006 Random selection of
    children
  • January-February 2006 Train on use of the ABAS
    II (if chosen) with liaisons, teachers, others
  • January-April 2006 Collect EI-entry child data
  • January-April 2006 Conduct file reviews to
    document program intensity
  • May 2006 Collect child progress data using ABAS
    II or program-chosen measure (each September and
    May)

36
TIME IN INTERVENTION
CHILD FAMILY PROGRESS OUTCOMES
PROGRAM INTENSITY
37
PEIOS LONGITUDINAL REPEATED MEASURES REGRESSION
RESEARCH DESIGN AND TIMELINE
REGION 1
September
May
REGION 2
REGION 3
ABAS II Program Scale D-SPECS P-SPECS
ABAS II Program Scale D-SPECS P-SPECS
38
What are the federal OSEP/ECO outcome indicators
and reporting timelines for PEIOS?
39
OSEP/ECO Child Outcome Indicator Domains
  • Positive social-emotional skills (including
    social relationships)
  • Acquisition and use of knowledge and skills
    (including early language/communication early
    literacy)
  • Use of appropriate behaviors to meet ones needs
  • Domains, sub-domains, and item content of
    measures are mapped to these integrated
    functional areas

40
Sample LaRosa Database
41
OSEP/ECO Child Outcome Indicator Metrics
  • of children who reach or maintain functioning
    level comparable to same-age peers
  • of children who improve functioning toward
    same-age levels
  • of children who did not improve functioning
  • of children maintaining own rate and preventing
    regression
  • of children showing specific curricular skill
    improvements compared to own previous skill
    levels
  • of children whose developmental progress
    profiles exceed own pre-intervention
    (maturational) expectations and those of their
    local EI peer group (IEI CEI PCI, HLM-EAPS)

42
Statistical Impact of ECI on Child Progress
Exceeding Maturation after 31 Months of
Programming Pooled HR/DD groups n 104
plt.000 48thilegt68thile 95 confidence
interval
43
OSEP/ECO Family Outcome IndicatorsDraft
Reconciliation
  • Based on ratings on the Family Outcomes Survey
    (in development)
  • Understand their childs strengths, abilities,
    and special needs
  • Know their rights and advocate effectively for
    their children
  • Help their children develop and learn
  • Have support systems
  • Access desired services, programs, activities in
    their community

44
Timelines for State Reporting of Child Outcome
Data to OSEP(Recent Report from OSEP/ECO
National Meeting , Washington, DC, 1/12-13/06
  • December 2005 SPP Measurement plan submitted
  • February 2007 APR Report on EI-entry child data
    onlyno progress data wanted
  • February 2008 APR 1st progress report
  • February 2009 APR 2nd progress report
  • February 2010 APR 3rd progress report

45
What is the collaborative model for training and
implementation in PEIOS and PAPREKA?
46
DIRECTOR
PAPREKA COORDINATORS Western
PA
Eastern PA
RESEARCH SYSTEMS DATA MANAGER
EVALUATION ASSISTANTS
STATISTICIAN PROGRAMMER
REGIONAL PA CONSULTANT
47
COUNTY MAWA SCHOOL DISTRICT PARTNERSHIP- Administr
ators
COORDINATORS
LIAISON
SPECS TEAM
REGIONAL CONSULTANT
TEACHERS CAREGIVERS
EVALUATION ASSISTANT
48
(No Transcript)
49
The Pennsylvania Pre-Kindergarten Analysis
(PAPREKA)A 4-Year Independent Program
Evaluation Research Collaborative to Document the
Impact and Outcomes of Partnership for Quality
Pre-Kindergarten (PQP)
50
SPECS Pennsylvania Early Childhood Intervention
Outcome Studieswww.uclid.org Early Childhood
Partnerships SPECS
  • Heinz Pennsylvania Early Childhood Initiatives
    (ECI) 1997-present) (Bagnato, etal., 2002 2005)
  • Pennsylvania Pre-Kindergarten Analysis (PAPREKA
    2005-2009 (Bagnato etal. 2005)
  • Pennsylvania Early Intervention Outcomes Study
    (PEIOS 2005-2008) (Bagnato etal., 2005)
  • TRACE Center of Excellence for Early Childhood
    Assessment (2002-2007 Dunst, Trivette, Bagnato)
  • The Efficacy of a Direct Instruction Add-On to a
    DAP Curriculum in 4KIDS at Braddock (2005-2007)
  • Pennsylvania Preschool Integration Initiative
    (PAPII 1989-1993) (Bagnato Neisworth, 1993)

51
The SPECS Team Evaluation of the
Heinz Early Childhood Initiatives
52
SPECS Evaluation of Heinz Pennsylvania ECI
Outcomes (1997-2004)
  • Nearly 4000 at-risk children
  • Urban and rural communities Pittsburgh, Erie,
    Central PA, York, Lancaster, Beaver
  • School district collaborations
  • Steady developmental gains
  • Delayed group gains
  • Social-behavioral gains, even for disorders
  • Early school success for gt400 children K-5
  • Reduced grade retention and special education
  • Program quality improvements
  • Improvements in parenting skills
  • Evolving community networking
  • Analysis of time-in-intervention effects
    (dosage)

53
What are the missions, research questions and
measurement design for PEIOS?
54
PAPREKA Missions for PQP
  • Document child and program outcomes attributable
    to PQP
  • Analyze the comparative impact of various school
    district/community ECE partnership models on
    child and program outcomes
  • Analyze the early school success of PQP children
    during their K year
  • Field-validate the PA Early Learning Standards
    and develop an ELS assessment rubric
  • Set the stage for future followup of PQP children
    into early grades and linkage with PSSA results

55
PAPREKA Research Questions for PQP
  • PQP children will show a pattern of actual
    progress which will outpace their maturational
    expectancies.
  • Program quality and interactive teaching styles
    will predict child progress and outcomes.
  • Certain types of partnership models will better
    predict child outcomes and program quality.
  • For both children at-risk and with delays, those
    who participate and remain engaged in the PQP
    programs for the longest periods of time
    (dosage) will show the most significant
    progress.
  • PQP children will demonstrate early school
    success in preschool and kindergarten based upon
    three criteria blind teacher assessments on the
    BSSI greater of ELS attainments and reduced
    grade retention and special education placement
    rates compared to historical district benchmarks.

56
TIME IN INTERVENTION
PARTNERSHIP MODEL
CHILDRENS EARLY SCHOOL SUCCESS OUTCOMES
PROGRAM ASPECTS
57
What is the authentic measurement approach and
timeline for PAPREKA?
58
Natural learning environment
  • ...the ongoing/routine/typical circumstances and
    contexts of a childthe childs natural
    developmental ecology that forms the
    environmental basis for real-life early learning
    and adaptation (OSEP, 1998).

59
SPECS Approach for PAPREKA
  • Design research model through collaboration with
    community partners (participatory action
    research)
  • Train teachers for ongoing child assessments
  • Conduct authentic assessments in everyday
    preschool settings using natural observations
  • Provide feedback to teachers parents for
    individualized early learning plans
  • Evaluate progress on Key Performance Indicators
    (KPI) child, program quality, teacher-child
    interactions, early school success outcomes
  • Classify type of partnership model in each site
  • Assess K achievements of children
  • Research impact of PQP

60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
PAPREKA Measures
  • Basic School Skills Inventory 3 (BSSI-3)
    (Hammill, etal, 1998)
  • Preschool and Kindergarten Behavior Scale (PKBS)
    (Merrell, 2003)
  • Early Childhood Environment Rating Scale
    (ECERS-R) Screener (Cassidy etal, 2005)
  • Teaching Styles Rating Scale (TSRS) (McWilliam,
    etal, 1996)
  • PA Early Learning Standards (ELS) Rubric (in
    development)

65
Sample MIS Database
66
A Sample of Marys Skills 22 months old
  • Some Things Im Really Good At
  • put 2 words together (examples 'Want more,' 'Get
    down')
  • refer to myself by name
  • participate in and enjoy nursery rhymes and
    finger games like 'Humpty Dumpty' or 'Itsy Bitsy
    Spider'
  • Some Things Im Working On
  • copy sounds made by others
  • drink from a regular cup without help
  • answer simple questions
  • Some Things I Want to Learn Next
  • dress myself, but I may need some help or
    supervision
  • share toys with other children when asked by an
    adult
  • count two or three objects (saying the number as
    I touch the object)

Database Merged Document
67
Class Snapshot 2Student Progress on Selected
Assessment Items
Highlighted Bolded Item s Areas of More
Program Support
68
(No Transcript)
69
Basic School Skills Inventory (BSSI)
  • Learning readiness skills for children
  • Authentic teacher observational ratings
  • Ages 48-107 months (Pre-3rd grade)
  • 6 Domains Spoken language Reading Writing
    Math Behavior Daily living
  • Standard and T-Scores (100/15 50/10)
  • Functional skills/benchmarks for learning
  • Graduated scoring 0, 1, 2, 3 (mastery)
  • Norms 757 children 5 states
  • PRO-ED

70
BSSI Subscale Samples
  • Spoken Language
  • Uses complete sentences when talking
  • Listens to and retells a story in sequence
  • Initiates and maintains conversations with others
  • Reading
  • Recognizes upper/lower case letters
  • Names letters when sounds are spoken
  • Has basic site vocabulary of 5 words

71
BSSI Subscale Samples
  • Writing
  • Writes from left to right
  • Writes first name without a model
  • Writes single letters when asked (b, h, m, t, a,
    e)
  • Mathematics
  • Counts objects in set of fewer than 10
  • Counts aloud from 1-20
  • Understands concepts of 1st, 2nd, 3rd

72
BSSI Subscale Samples
  • Classroom Behavior
  • Makes friends easily
  • Takes turns
  • Uses teacher feedback to improve learning
  • Can attend to activity for 5 minutes
  • Daily Living Skills
  • Enters and exits school by self
  • Assumes responsibility for own belongings

73
BSSI Rating Scale
  • When completing the BSSI, a four-point rating
    scale is used to rate each behavior or skill
  • 0 (Does not perform)
  • 1 (Beginning to perform)
  • 2 (Performs most of the time)
  • 3 (Performance indicates mastery)

74
National Normative Comparisons for ECI Childrens
Early Learning Abilities Second Semester Grades
K-1 Basic School Skills Inventory-Revised
Mean Standard Score
Reading
Math
Daily living skills
Social skills
Academic competence
Domain
75
Acquisition of Kindergarten Precursor Skills
Matching PA Early Learning StandardsAchievement
in Final 6 Months of ECI
Learning Skill s Achieved
76
Statistical Impact of ECI on Child Progress
Exceeding Maturation after 31 Months of
Programming Pooled HR/DD groups n 104
plt.000 48thilegt68thile 95 confidence
interval
77
(No Transcript)
78
Preschool and Kindergarten Behavior Scales (2003)
  • 36-72 months
  • Dual focus on social skills and problem behaviors
  • Competencies/concerns with treatment validity
  • Cooperation, interaction, independence, and
    behavior domains (attention, withdrawal)
  • Parent and professional ratings
  • Norms 2,855
  • Graduated scoring and SS
  • PRO-ED

79
Preschool and Kindergarten Behavior Scales (2003)
  • SOCIAL SKILLS Never Rarely
    Sometimes Often
  • Works or plays independently 0
    1 2 3
  • Follows instructions from
  • adults
    0 1 2 3
  • Shows self-control 0
    1 2
    3
  • Participates in family or
  • classroom discussions 0
    1 2 3
  • Follows rules
    0 1 2
    3
  • Takes turns with toys and
  • other objects
    0 1 2
    3

80
Preschool and Kindergarten Behavior Scales (2003)
  • PROBLEM BEHAVIOR Never Rarely
    Sometimes Often
  • Acts impulsively without
  • thinking
    0 1 2
    3
  • Must have his/her own way 0
    1 2 3
  • Is restless and fidgety 0
    1 2 3
  • Withdraws from the company
  • of others
    0 1 2
    3
  • Is overly sensitive to criticism
  • or scolding
    0 1 2
    3
  • Disrupts ongoing activities 0
    1 2 3

81
Social-Behavioral Progress Pattern for ECI
Children Behavior Disorder Group (18) T1-T5
Mean Standard Score
Domain
82
Roles for Authentic Assessment in PAPREKA
  • Teachers
  • September and May
  • (Note January for 2006)
  • BSSI-3
  • PKBS
  • PAPREKA Team
  • September and May
  • (Note January for 2006)
  • ECERS-R Screen
  • TSRS

83
Linking Assessment to State Early Childhood
OutcomesPA Early Learning Standards (ELS)
  • Approaches to Learning
  • Creative Arts
  • Language Literacy
  • Logical-Mathematical
  • Personal-Social
  • Program Partnerships
  • Science
  • Social Studies
  • Demonstrate initiative curiosity
  • Expresses self through movement music
  • Understands language sounds
  • Develops space-shape concepts
  • Develops self-control skills
  • Form school partnerships with ECE
  • Acquire knowledge about matter and living things
  • Understand role of self in community

84
Linking BSSI Items to the PA Early Learning
Standards
BSSI Area or Domain BSSI Item PA EL Standard (for 4- 5-year-olds)
Spoken Language Answers questions correctly after listening to a story SL 11 Develop and expand listening and understanding skills RL 1
Reading Follows printed instructions (reads and follows directions) R 20
Classroom Behavior Demonstrates a readiness to respond when his or her turn comes during group activities CB 11
Daily Living Skills Follows directions relating to paper-and-pencil tasks DL 10
85
Linking BSSI Items to the PA Early Learning
Standards
BSSI Area or Domain BSSI Item PA EL Standard (for 4- 5-year-olds)
Mathematics Assigns the correct numeral to a set of objects M 7 Learn about numbers, numerical representation, and simple numerical operations LM 1
Understands the concepts of first, second, and third M8
Daily Living Skills Tells time within 5 minutes from analog watch or clock face DL 19

86
Overview of Programmatic Measures ECERS TSRS
  • ECERS-R Screener
  • 16 core items
  • Two factors
  • Activities-Materials (9)
  • Language-Interaction (7)
  • TSRS
  • 20 items
  • Focus on teacher-child interaction during
    instruction
  • 8 teaching strategy variables Redirects,
    Introduces, Elaborates, Follows, Informs,
    Acknowledges, Praises, Affect

87
PAPREKA LONGITUDINAL REPEATED MEASURES REGRESSION
RESEARCH DESIGN AND TIMELINE
PARTNERSHIP MODEL 1
September
May
PARTNERSHIP MODEL 2
PARTNERSHIP MODEL 3
BSSI-3 PKBS ECERS-R TSRS
BSSI-3 PKBS ECERS-R TSRS
88
Roles of Implementation Sites in the Partnership
Implement Assessment Gain Incentive Funds
  • Designate PQP Evaluation Liaison
  • Train with PAPREKA team to learn assessments
  • Train teachers in assessments
  • Gain signed parent consent
  • Ensure completion of assessments by teachers
    2x/year
  • Collect completed assessment forms and transfer
    hard copies to PAPREKA Coordinator
  • Show teachers how to use feedback letters

89
Roles of PAPREKA Team for Implementation Sites
  • Meet with partnership to form trusting
    collaboration
  • Maintain confidentiality
  • Train liaison and teachers in authentic
    assessment best practices
  • Complete programmatic observations 2x/year
  • Collaborate to classify type of partnership in
    each program
  • Produce child feedback letters
  • Issue 1 year-end report card
  • Analyze aggregate data on child outcomes

90
PAPREKA Team Roles with The PQP Planning Sites
  • Schedule collaborative meetings to develop our
    partnership early as a basis for future work
  • Offer early training sessions on assessment
    purposes and eventual evaluation model and
    process
  • Demonstrate computer child feedback options for
    programs to guide teaching and quality
    improvements
  • Consult with partnerships to facilitate their
    eventual implementation proposal
  • Collect extant information from sites on various
    programmatic features
  • Begin process of classifying sites by type of
    partnership model

91
Not everything that can be measured counts, and
not everything that counts can be measured
(Einstein, 1951)
92
The MisMeasure of Man (Stephen J. Gould, 1981)
  • We pass through this world but once. Few
    tragedies can be more extensive than the stunting
    of life, few injustices deeper than the denial of
    an opportunity to strive or ever hope, by a limit
    imposed from without, but falsely identified as
    lying within (p.28).
Write a Comment
User Comments (0)
About PowerShow.com