Now that you have data, what do you do with it

1

You Have the Assessment Data Whats Next?
2

• Pittsburgh Public Schools
• Dr. Thekla Fall, Program Officer
• Ms. Susan Cefola, Resource Teacher
• www.pps.k12.pa.us/AcademicOffice/WorldLanguages/
• Prismatic Consulting LLC
• Mr. Chris Dalessandri, Managing Partner
• www.PrismaticConsulting.Com
• www.OWLTS.Com

3

• Do you speak the language of your administrators?

4
Basic Beliefs

• Teachers
• Testing drives instruction
• Students
• Testing drives learning
• Administrators
• Data-driven decision making

5
Basic Beliefs
6
What is useful/significant data that
will help drive instruction/learning/
decision making?

• National standards
• are what students should know and be able to do
  to graduate from high school
• are for all modern languages
• should be assessed using the same measurement
  tool across languages resulting in data showing
  growth over time (formative as well as summative
  assessments)

7
The PPS ORALS test is a proficiency test based
on the SOPI /OPI!

• Interpersonal Mode of the Communication Standard
• Can be used with all modern languages
• Uses a nationally known measurement
• Can be used across grade levels

8
To understand the datawe must understand the
TEST!
9
To understand the datawe must understand the
TEST!

• Achievement vs. Proficiency Test

• My kids in level 2 are Intermediate Mid on the
  ACTFL Scale!

• The test should go along with the text I am
  using!

My kids did well on the chapter test -these
ratings are wrong!
10
To understand the datawe must understand the
TEST!
Achievement Test? OR Proficiency Test?

• Chapter/Unit tests
• Portfolio Assessments
• ACTFLs Integrated Performance Assessments (IPAs)
  

• ACTFL OPI
• Simulated Oral Proficiency Interview (SOPI)
• Computerized Oral Proficiency Instrument (COPI)
• PPS ORALS

11
To understand the datawe must understand the
TEST!

• Teachers as Raters
• Better grasp of the ACTFL Scale
• Understand the finer points of what students are
  asked to do on the test
• Hear the problems students have in responding
• If testing is to drive instruction teachers need
  to really understand the test

12
PPS ORALS Online Test

• Assesses students in grade 5, grade 8, level 3
  high school, and seniors
• Two levels of assessment
• Proficient Level (to Intermediate Low)
• Distinguished Level (to Intermediate High)
• Same tasks across languages
• Rate speech samples using the ACTFL Scale
  resulting in a proficiency rating
• Ratings are done online by PPS teachers

13
Focus Assessment on Key Items

• Stick to the IMPORTANT items

Create sentences
Survival Topics
Give directions
Ask Questions
Present narration
Give simple greetings
PPS ORALS Proficient Level Test
14
Focus Assessment on Key Items

• Stick to the IMPORTANT items!

Expanded Survival Topics
Paragraph level discourse
Complex sentences with connectors
Past and present tenses, future time
Past Narration Description

Situations with a complication

PPS ORALS Distinguished Level Test
15
Announcing

• The NEWEST version of the PPS ORALS assessment
  software

16

What gets counted, counts!

• Without data world language programs often
• are invisible mute
• are subject to changes without heed to
  consequences
• Data provides tangible facts and figures that
  make it harder to ignore world language programs
• On the other hand

Grant Wiggins, Schenley High School, 1993
17
We need to stop linking data with accountability
and focus on program improvement.
Data often has an inherent FEAR FACTOR when
someone says
Assessment DATA!!!

18
Basic questions about data!

• Is it reliable?
• Is it valid?
• Is it useful?
• All measurement has some error-
• how much is acceptable?

BCTF Student Assessment and Evaluationgt A Primer
on Interpreting Educational Data
www.bctf.bc.ca/education/assessment/EducationalDa
taPrimer
19
Using the data

• Start quietly
• Check Validity and Inter-rater reliability
• Make sure data is useful
• Less is more--dont flood people with data
• Focus on the important/significant data
• Present data in context
• Provide support
• Other Uses of Data

20
Start quietly

• Give yourselves and your teachers time to learn
  there is no need to broadcast the results to the
  world right away.
• Use the preliminary data to set benchmark levels
  (where should students be- at what level)
  depending on your program features and overall
  goals.
• Give yourselves time to develop materials and
  provide training to improve scores if needed.

21
Check Validity Inter-Rater Reliability

Validity--About 80
70 of PPS ratings in the sample were equal or
differed by only one minor border.
22
Make sure the data is useful!

• ASK
• Is it significant / important data?

Will the data help improve programs?
Will the data help improve instruction?
Percentage of students who can state their name
in French from 2003-2005!
23
Make sure the data is useful!
Are students attaining benchmark and standard
levels by district, school, class?
24
Make sure the data is useful

• Assessment results from 3 programs magnet,
  recommended standard, standard

What types of programs result in a high level of
proficiency?
25
Make sure the data isuseful
Looking at sub-groups--how proficient are
students by gender, race, socio-economic status,
and attendance?
26
Make sure the data isuseful
What impact do new programmatic features have on
student proficiency?
27
Less is more

• Manageable data how much can administrative
  staff and teachers use effectively?
• Who should see what data?

3 ½ Inch Binder
28
Present the data in a context!PPS Middle School
Program Profiles
29
Present the data in a context!PPS Middle School
Program Profiles
PSMLA Exemplary Program (PEP) Awards / Statewide
(www.psmla.org)
30
Provide supportRating Data Problems with
inter-rater reliability

• Support provided
• Online Rater Calibration Practice for Teachers
• Rater comments that provide feedback to students
  and raters
• Y Connectors for headsets so that 2 teachers can
  rate together-- pair experienced rater with new
  rater

31
Provide supportSurvey DataTeachers are not
using instructional support materials
Although I did not use it, I was aware of
what it contained (is this the same thing?) and
therefore structured my calendar to include
abilities it reflected.
It's an easy tool for kids to look at in Sept or
Oct. and think I don't know that and by 2nd
semester they can look down and say Hey, I know
that now.
I only used it with the eighth grade.
Cumbersome to use in a large class if info is
really to be checked.

• Instructional support provided
• In-service training and in-class modeling
• 4 Checklists based ACTFL Scale
• PPS Speaking Rubric / refinements

32
Provide supportClass data Students need
more practice asking questions

• Instructional support provided
• Situations for Communication
• Online Practice Activities for Students (Sound
  recorder / PALS)

33
Provide Support Helping Students/Parents
understand the data

• Guide for Students and Parents
• Rubric / Checklists
• Posters in every classroom
• Certificates not only state the rating but also
  explain
• the significance of the rating

34
Provide SupportStudent Certificate

• Congratulations!
• You have exceeded the PPS World Language
  Communication (Speaking) Standard goal by
  attaining the Intermediate High Level! You are to
  be commended!
• To attain this level, you have demonstrated on
  the Online PPS ORALS that you can
• Sustain a conversation about many different
  concrete topics.
• Converse more than 50 of the time in
  paragraph-like connected speech.
• Narrate and describe using past tenses and future
  time more than 50 of the time.
• Be understood even by someone who is not
  accustomed to dealing with language learners.
• To reach the next level of the standard, the
  Advanced Low Level (the level needed for world
  language teacher certification on the ACTFL
  Scale), you must show that you can
• Sustain a conversation with a complication.
• Converse in paragraph-like connected speech.
• Narrate and describe using past and future
  tenses.
• The American Council on the Teaching of Foreign
  Language (ACTFL) Scale is used across the US to
  rate speaking proficiency. Many colleges and
  universities use the ACTFL Scale. The School
  District of Pittsburgh uses the national ACTFL
  scale as the basis for the Online PPS ORALS
  assessment and rating. The following chart
  compares Pittsburgh School District terms to
  ACTFL terms.
• Superior (Graduate School)
  
• Advanced (College Level)
  
• Intermediate High (College Level)
  
• Intermediate Mid (College
  Level)

35
Other Uses College credit for students

• ACTFL ACE Credit

36
Other UsesPA Department of Education

• PA World Language Teacher
• Certification currently requires
• Intermediate High Proficiency!

Teachers should at least be at the ACTFL Advanced
Low Level!
37
Using Data

• Sometimes the earnest discussions about
  standards, setting appropriate performance levels
  and benchmarks, are more important and more
  useful than the data that generated the
  discussion.

BCTF Student Assessment and Evaluationgt A Primer
on Interpreting Educational Data
www.bctf.bc.ca/education/assessment/EducationalDa
taPrimer