A New Vision for 21st Century Education

1
Innovations for Assessing 21st Century Skills
A New Vision for 21st Century Education
Panel Presentation on Assessment of 21st Century
Skills CCSSO National Conference on Large Scale
Assessment San Francisco, California June 27, 2006
Insert Presenter Name Insert Presenter Title
Company Insert Event Name Insert
Date PLEASE NOTE This is only a template
presentation you may add examples and additional
slides based on your audience EDUCATION COMMUNITY
AUDIENCE
2
Overview
Key Message
In order to produce students with 21st century
skills, we need to assess 21st century skills.
3
Overview

• Overview and Introduction Ken Kay
• Who is the Partnership
• Why are 21st Century Skills so critical?
• What is the framework for 21st Century Skills?
• Why is the assessment of 21st Century Skills so
  important?
• What are the potential ways to assess 21st
• Century Skills Kathy Comfort, Steve Klein,
  
• and Terry Egan
• III. What are some general observations about
  21st
• Century Skills and Assessment Jim Popham

4
Who is the Partnership?
5
Who is the Partnership?
6
21st Century Skills
Why are 21st Century Skills so Critical?
5 Reasons
7
21st Century Skills

• 1. We need our students to become effective
  21stCentury Citizens.

8
21st Century Skills
2. The U.S. is falling behind.
9
21st Century Skills
Ranking of G8 countries 10th grade math
problem solving
OECD Ranking
Problem Solving
Math
Science
Reading
14th
15th
15th
18th
18th
24th
24th
2000
2000
2000
2003
2003
2003
2003
Source PISA, 2000, 2003
Courtesy of Cisco
Systems
10
21st Century Skills
3. The magnitude of our competition is changing.
11
21st Century Skills
China India
300 Million Skilled Workers
Japan
25 Million Skilled Workers
2025
1985
12
21st Century Skills
4. The nature of work is changing.
13
21st Century Skills
How many of your Parents Grandparents had only
one or two jobs in their lifetimes?
14
21st Century Skills
How many jobs will a young person have today
between age 18-38?
10.2 jobs
15
21st Century Skills
21st Century
20th Century
1 2 Jobs
10 15 Jobs
Number of Jobs
Flexibility And Adaptability
Mastery of One Field
Job Requirement
Integration of 21st Century Skills into Subject
Matter Mastery
Subject Matter Mastery
Teaching Model
Subject Matter Mastery
Integration of 21st Century Skills into Subject
Matter Mastery
Assessment Model
16
21st Century Skills
5. To help all children reach higher levels of
academic achievement.
17
What is the Framework for 21st Century Skills?
18
21st Century Skills Framework
20th Century Education Model
19
21st Century Skills Framework
20
21st Century Skills Framework
Core Subjects

• - English
• Reading or Language Arts
• Mathematics
• Science
• Foreign Languages
• Civics

• Government
• Economics
• Arts
• History
• Geography

21
21st Century Skills Framework
Thinking and Learning Skills

• Critical Thinking Problem Solving Skills
• Creativity Innovation Skills
• Communication Information Skills
• Collaboration Skills
• Contextual Learning
• Information and Media Literacy

22
21st Century Skills Framework

• ICT Literacy
• Information and communications technology (ICT)
  literacy is the ability to use technology to
  accomplish
• thinking and learning skills
• Critical Thinking Problem Solving Skills
• Creativity Innovation Skills
• Communication Information Skills
• Collaboration Skills
• Contextual Learning
• Information and Media Literacy

23
21st Century Skills Framework

• Personal Skills
• Leadership
• Ethics
• Accountability
• Adaptability
• Productivity
• Personal Responsibility
• People Skills
• Self Direction
• Social Responsibility

24
21st Century Skills Framework

• 21st Century Content
• Global Awareness
• Financial, Economic, Business and
  Entrepreneurship Literacy
• Civic Literacy
• Health Wellness Awareness

25
21st Century Skills Framework

• The Design Specs for 21st Century Education
• Core Subjects
• Thinking Learning Skills
• ICT Literacy
• Personal Skills
• 21st Century Content

26
Why is Assessment of 21st Century Skills so
important?
27
Assessment
Accountability and metrics are here to stay.
But, what we measure matters!
28
Assessment
If we want 21st Century innovative learners we
need to measure metrics of innovative learning!
29
Assessment
Assessment of 21st Century Skills The
Current Landscape June 2005 Partnership for
21st Century Skills www.21centuryskills.org
30
Conclusion

• Every student in this nation must be
• An analytic thinker
• A problem solver
• Innovative and Creative
• An effective communicator
• An effective collaborator
• Information and media literate
• Globally aware
• Civically engaged
• Financially and economically literate

31
Conclusion
In order to assure our student possess 21st
century skills, we need to assess 21st century
skills.
32
PASS
The Partnership for the Assessment of
Standards-based Science CCSSO Large-scale
Assessment Conference Partnership for 21st
Century Skills Panel San Francisco, CA June 27,
2006 Kathy Comfort, WestEd
33
PASS The Partnership for the Assessment of
Standards-Science

• Developed in 1996 with NSF support
• Test design and specifications grounded in
  research
• Aligned to the National Science Education
  Standards
• Contains different types of measures including
  performance tasks
• Assess what is valued (not just easily measured)
• Appropriate for grades 4-12
• Students in 22 states and Puerto Rico have been
  assessed with PASS

34
The PASS Assessments and the 21st Century
Framework
35
PASS Assessment Components
36
Sample Performance Task
37
Sample Performance Task
38
Sample Performance Task
39
Sample Performance Task
40
Sample Performance Task
41
Sample Performance Task
42
Sample Performance Task
43
Sample Performance Task
44
Sample Performance Task
45
PASS Current and Future

• PASS Current
• Students record responses in test booklets
• Scored by independent trained readers

• PASS Future
• Technology-based pilot found it is feasible to
  use computers to deliver the tasks to students
  and to
• machine score their open-ended answers
• Simulations (virtual labs)

46
PASS Assessments are Technically Sound

• Valid - consensus on correspondence to standards
• Statistical analysis have found PASS scores to be
  reliable and sensitive to the effects of
  instruction
• High inter/intra reader consistency
• Disaggregated data for gender and ethnic/racial
  groups

47
The Real Deal
Feasibility Yes--it can be done as part of a
large-scale assessment, and it has.

• However
• It is not easy to develop valid and reliable
  tasks. Pilot testing takes on a whole new
  meaning.
• - If you cant get pumice to float neither
  will the
• 10,000 teachers administering the
  test.
• You need a lot of stuff to do performance
  assessment. (The stuff can be expensive and
  not easily found.)
• Hand scoring takes time and readers must be
  highly trained and calibrated for reliable
  results.

48
Contact Information
Kathy Comfort PASS at WestEd WestEd 730 Harrison
St. San Francisco, 94107 415-615-3161 or
kcomfor_at_wested.org
49
Collegiate Learning Assessment (CLA)

• Stephen KleinCouncil for Aid to Education (CAE)
• June 27, 2006

50
Overview

• Five principles driving the CLA
• CLAs distinguishing features
• CLAs measures
• Reporting results
• Program participation

51
Five Principles Driving the CLA

• One test cannot assess overall quality
• Can measure some important abilities
• Need benchmarks to assess progress
• Use results appropriately
• Use psychometrically sound tests

52
CLAs Distinguishing Features

• Use open-ended tests that are
• Realistic work samples Typical performance
• Engaging and psychometrically sound
• Focused on high level abilities
• Appropriate across schools and majors
• School is unit of analysis
• Sample students within schools
• Matrix sample tasks
• Paperless administration, scoring, reporting
• Benchmark schools results to comparable
  institutions to provide value-added metric

53
CLAs Measures

• Analytic writing (essay) prompts
• Make-an-argument (45 minutes)
• Break-an-argument (30 minutes)
• Performance tasks (90 minutes)
• Document based
• Contextualized questions
• Split screen/dialogue box format
• Analytic and holistic scoring
• Background questionnaire

54
Make-An-Argument Prompt

• In our time, specialists of all kinds are
  highly overrated. We need more generalists
  people who can provide broad perspectives.
• Directions 45 Minutes, agree or disagree and
  explain the reasons for your position. Student
  selects one of two prompts to answer.

55
Break-An-Argument Prompt

• Students are asked to discuss how well reasoned
  they find an argument to be (rather than simply
  agreeing or disagreeing with it).
• A respected professional journal with a
  readership that includes elementary school
  principals published the results of a two-year
  study on childhood obesity. This study sampled 50
  children, ages 5-11, from Smith Elementary
  School. A fast food restaurant opened near the
  school just before the study began. After two
  years, students who remained in the sample were
  more likely to be overweightrelative to the
  national average. Based on this study, the
  principal of Jones Elementary School decided to
  address her schools obesity problem by opposing
  the opening of any fast food restaurants near her
  school.

56
Performance Tasks

• 90-minute real life problems
• General directions and context
• Need to combine information from different
  types of documents
• A few open-ended questions, no one right
  answermust explain rationale
• Split screen
• Right Document Library
• Left Question and answer block

57
Swiftaire 235

• You advise the president of DynaTech
• DynaTech makes precision airplane navigation and
  communication instruments
• DynaTechs sales manager suggests buying a
  Swiftaire 235 to visit customers and demo (and
  show confidence in) its products
• Recent accident wing fell off in flight
• Students tasks
• Review document library
• Write memo discussing pros and cons
• of DynaTech getting a Swiftaire 235
• Justify your recommendations

58
Document Library

• Newspaper article about accident
• Federal report about in-flight breakups
• DynaTech email exchanges regarding reasons to buy
  or lease a Swiftaire 235
• Excerpt from trade magazine article that compares
  Swiftaire 235s performance and safety
  characteristics to similar planes
• Manufacturer specifications and required pilot
  training for Swiftaire 180 and 235

59
Scoring Rubric

• Writing skills clear, organized, persuasive
• Analysis, problem solving, reasoning skills
• Integrates information from different sources
• Recognize flaws and issuesnot swayed by
  emotional arguments, faulty logic, irrelevant
  information, etc.
• Anticipates consequences and implications of
  alternative solutions and strategies
• See pros and cons of competing explanations,
  points of view, and arguments
• Weighs evidence based on its credibility

60
Variety of Performance Tasks

• Airplane
• Brain Boost
• Catfish
• Crime
• Expedition
• Lake to River
• Museum
• Parks
• Skating

61
Psychometrics

• Median inter-reader correlation .82
• Internal consistency reliability
• Student level .80
• School level .97
• Student level correlations with SAT
• PT score .56
• Essay score .44
• School level correlations with SAT
• PT score .92
• Essay score .79
• Total battery .88
• N 15,000 students, 113 schools

62
Two Definitions of Value-Added

• Improvement within an institution over time
  (e.g., seniors score higher than freshmen with
  the same SAT scores)
• Improvement between classes within an institution
  relative to the improvement between classes at
  other institutions
• Two types of benchmarks/reference groups
• ACT/SAT scores only
• Peer Group (school characteristics)

63
Fig. 1 Relationship Between Mean ACT Scores and
Mean Total CLA Scores for Freshmen
31
Your Institution (Freshmen) Others (Freshmen)
27
CLAScore
23
Regression Intercept 8.02 Slope 0.66 R-square 0.80
19
15
15
19
23
27
31
ACT Score
64
Fig. 2 Relationship Between Mean ACT Scores and
Mean Total CLA Scores for Seniors
31
Your Institution (Seniors) Others (Seniors)
27
CLAScore
23
Regression Intercept 11.96 Slope 0.62 R-square 0.7
5
19
15
15
19
23
27
31
ACT Score
65
Fig. 3 Relationship Between Mean ACT Scores and
mean Total CLA Scores for Freshmen and Seniors
31
27
CLAScore
23
19
15
15
19
23
27
31
ACT Score
66
Percentage of Schools by Carnegie Classification
IPEDS(N 1421)
CLA(N 115)

• Carnegie Classification
• Doctoral/Research Universities - Extensive 11 11
• Doctoral/Research Universities - Intensive
  8 11
• Masters Colleges Universities I 35 37
• Masters Colleges Universities II 8 3
• Bachelor Colleges Liberal Arts 16 19
• Bachelor Colleges General 23 17

67
Program Participation

• School Year Institutions Students
• 2003-04 46 2,040
• 2004-05 57 8,009
• 2005-06 Fall Spring 121 gt30,000

68
Frequently Asked Questions

• Does maturation affect score gains between
  classes?
• Does student motivation affect scores?
• How do you control for selection effects?
• Is there an interaction between academic major
  and prompt type?
• Why not use portfolios or graduate or
  professional school admissions tests?

69
Thinking Beyond TechnologyPartnership for 21st
Century Skills PanelCCSSO ConferenceSan
Francisco, CAJune, 2006Teresa EganEducational
Testing Service
ICT Literacy Assessment
70
ICT Literacy A Bridge

• Information and Communication
• Technology Literacy
• Can I find information on the web?
• Can I create a persuasive presentation?

Information Literacy

• Technical Literacy

Database
Word Processing
Presentation
Access
Evaluate
Use

• Can I bold a word?
• Can I open a database?

• Can you find information?
• Can you evaluate authority?

71
Why are ICT Literacy skills an important
component of a 21st century model for education?

• Being ICT literate impacts the way we live,
  learn, and work in todays information society.
• Technical skills are not enough. To be successful
  citizens and productive workers in the 21st
  century, we must be able to apply technical
  skills to cognitive tasks retrieving,
  synthesizing and communicating information
  effectively.
• There is a lack of information about the ICT
  literacy skills of students, and debate about how
  best to address this issue in academic curriculum.

72
What were the challenges in developing an
innovative measure of ICT Literacy proficiency?

• Design an assessment with face validity
• Authentic, relevant, performance-based
• Measure higher-order thinking skills in a digital
  environment
• Include ethical and legal considerations in the
  use of information
• Ensure that it is based on solid measurement
  principles
• Quality
• Validity
• Reliability
• Provide both students and institutions with
  useful data and feedback

73
Where to Start?

• 2001 Convene International Panel
• Australia, Brazil, Canada, France, U.S.
• Include representatives from business, education,
  and governmental organizations
• Results Produced Digital Transformation a
  framework for ICT Literacy, available at
  www.ets.org/ictliteracy
• Design an assessment based on this construct and
  the Association of College Research Libraries
  (ACRL) Information Literacy Competency Standards
  for Higher Education (January, 2000)
• Engage subject matter experts from IHEs to
  develop and pilot the ICT Literacy Assessment

74
Where to Start?
Define
ICT Literacy Using cognitive skills in a
technological environment
Access
Manage
Integrate
Evaluate
Create
Communicate
75
Basic Design Features

• Two levels of difficulty
• 75 minutes in length
• Interactive tasks using simulated software with
  look and feel of typical applications (databases,
  search engines, spreadsheets, email, etc.)
• 3-5 minute and 15-minute tasks
• Real-life scenarios
• Get back on track mechanisms
• Multiple scorable elements per task
• Online score reporting

76
ICT Literacy Assessment Content

• Content Areas
• Humanities
• Social Sciences
• Natural Sciences
• Practical Affairs
• Popular Culture
• Contexts
• Academic
• Business
• Personal

• Technology Tools
• Word Processor
• Presentation Software
• Email
• Database Search Engine
• Web Browser/Search Engine
• Spreadsheet or Table
• Graphing Software
• File Manager
• Electronic Bulletin Board
• Instant Messenger

77
Sample Score Report
78
What have we accomplished to date?

• Administered two versions of the assessment
  large scale and individual
• Two levels of difficulty
• Advanced
• Core
• Over 10,000 students tested at 68 institutions

79
What are some of the findings from this initial
testing?

• When selecting a research statement for a class
  assignment, 25 of test takers picked statements
  that did not address the assignment.
• When asked to narrow an overly broad search, more
  than 80 of students could not correctly describe
  the specific problem with the initial search.
• When asked to evaluate a set of websites, 48 of
  test takers identified the website that met the
  criteria of currency, authority and objectivity.

80
Student feedback from first administration
81
Student Comments

• I feel that I probably did better on the tasks
  that dealt
• with things that I use regularly email and
  instant
• messaging.
• I do not think that I am the most ICT literate
  person
• in the world. I love to use computers and
  other
• technologies for recreational uses, and see
  the vast
• benefits of their other uses as well, I just
  have not
• totally used them to their potential.
• I had trouble with most of the test, but I
  expected this.
• I need help getting started in this new
  technology.
• All around, I think it's ok, but like my skills,
  there's
• room for improvement.

82
What have been our biggest impediments to date?

• Getting buy-in on the importance of measuring
  ICT Literacy funding!
• Who owns the responsibility for ICT Literacy on
  campus? Faculty? Librarians? Administrators?
• Delivery challenges computer labs, student
  recruitment, proctors, technical issues
• Interpreting and utilizing data to effect changes
  in instructional design
• Implications evident for professional development
  needs

83
Next Steps Ongoing Validity Studies

• Comparison with self-report measures
• Alignment with expectations of employers
• Expert review of assessment design, tasks, and
  scoring
• Cognitive strategies on ICT literacy assessment
  and naturalistic tasks
• Educational outcomes of ICT literacy instruction
• Comparison with writing portfolios

84
Contact Information

• For More Information
• Pick up a CD for demo tasks and more information
• Visit the ETS booth for product demo
• Visit our website
• www.ets.org/ictliteracy

85
Contact Us
The Partnership for 21st Century Skills 177 North
Church Avenue, Suite 305 Tucson, AZ 85701 (520)
623-2466 www.21stcenturyskills.org