Characteristics of an Effective Assessment System

1
Characteristics of an Effective Assessment System
David C. Smith, Dean Emeritus College of
Education University of Florida smith_david_c_at_hotm
ail.com http//education-assessment.org/ Ph.
563-382-0485
2
A Definition

• A carefully designed and functional, automated
  system for collecting, organizing, maintaining,
  analyzing and utilizing meaningful information.

3
What It Is Not

• A collection of currently existing data sets.
• A collection of data dealing exclusively with
  candidate performance.
• An assessment system designed to meet the needs
  of an accrediting agency.
• Does not contain data on the performance of
  individual candidates.
• Does not contain data on the performance of
  individual faculty.

4
What It Is

• First and foremost, an assessment system serves
  the needs of the unit for data to contribute to
  decisions that lead to the improvement of the
  effectiveness and efficiency of the unit.

5
Know Where You Are Going Before You Start on a
Journey

• There are often situations where existing data
  are assembled and anointed as an assessment
  system.
• This results in inevitable gaps and unintended
  redundancy.
• The assessment system needs a framework and a
  structure.
• Framework Design
• Structure Levels

6

• Have a Clear Vision of What You Want Your
  Assessment System To Do

7
Serve a Variety of Needs for Data

• Assessing programs
• Improving programs
• Allocating resources
• Acquiring resources
• Generating resources
• Demonstrating effectiveness

• NCATE
• Program approval
• Title II
• PEDS
• Institutional review
• Regional accreditation
• Board of Regents review

8
A Critical Question

• A set of assessment systems, each for a single
  purpose.
• Or
• A single assessment system designed to serve
  multiple purposes.

9
An Assessment SystemRequires Resources

• Personnel
• Someone to coordinate the effort
• A committee to assist in the design of the system
• Space
• Space allocated to house the system
• Hardware and Software
• These needs are minimal

10
A Critical Characteristic

• The assessment system should adapt to the data
  within the unit.
• The unit should not have to adapt to the
  assessment system.
• The assessment system should incorporate data
  meaningful to the unit.

11
Phases of Putting an Assessment System in Place

• Designing
• A critical and often under emphasized phase
• Design the system to fit your needs for data
• Who to involve in the design
• Otherwise the system is not likely to be used
• Inevitable tension Comparison and Uniqueness
• Implementing
• Import meaningful data
• Generate data that otherwise would not exist
• Maintaining
• A less expensive phase
• Evaluating and Revising
• Eliminate data that are not useful
• Identify potentially valuable that are not in the
  system

12
Flexibility and Adaptability

• The assessment system should adapt to the needs
  of the unit.
• An effective assessment system should not require
  the users to adapt the data they collect to fit
  the system.
• The assessment system should be sufficiently
  flexible to be reconfigured to meet the demands
  of the unit.

13
The Focus of the Assessment System Should be on
the Unit

• It should deal with aggregate and central
  tendency data.
• It should be able to generate data beyond that
  which in entered into the system. (Value added)
• It should deal primarily with unit data,
  institutional data, comparative data and trend
  data.
• It should be able to incorporate levels of data.
• The system should facilitate data based decision
  making and program improvement. (Not individual
  candidate and faculty assessment.)

14
Collects and Generates Data More Powerful than
Anecdotal Data

• Aggregate data is more powerful than anecdotal
  data because generalizations can be made about
  groups rather than individuals.
• Aggregate candidate data Program data

15
Interaction Among Different Kinds of Data

• Aggregate candidate data
• Aggregate faculty data
• Resource data
• Productivity data

16
Contributes to Analysis

• The assessment system should be useful in
  generating data in the form of templates.
• Graphs
• Tables

17
Aggregate Candidate Performance
18
A Calendar for Entering Data

• A calendar for collecting the data is useful.
• The timetable when various data are available is
  varied.
• A calendar for reporting the data is useful.
• The times at which the data are needed varies.
• The use of calendars makes the workload more
  evenly distributed.

19
A Calendar for Reporting Data

• A calendar specifying when data are needed is
  useful. (PEDS, budget building, Title II.)
• A calendar specifying when data become available
  is useful. (Enrollment, grades, fiscal.)
• Color coding the calendar is helpful.

20
Proactive Data The Creative Dimension

• What information do others ask for?
• What information do they need?
• How important is what you do?

21
Describe an Effective Assessment System

• It collects and generates useful information.
• It provides data that contributes to the
  improvement of programs.
• It is built on a design and structure.
• It serves multiple needs for data.

22
Ends and Means

• It is essential that the assessment system be
  incorporated into the regular operation of the
  unit.
• Otherwise it is likely that the data will not be
  regularly generated and the system will fall into
  disuse.
• Accreditation expectations call for an assessment
  system that is maintained over time.

23
The Central Question

• How does, or will, your assessment system
  accommodate the characteristics of an effective
  assessment system?

24

• For more information see
• http//education-assessment.org/