For Data Geeks

1
For Data Geeks
2
Connecting Assessment with Practice Moving
Information fromInteresting to Valuable

• Darlena Jones, Ph.D.
• Director of Research and Development
• Educational Benchmarking Inc

3
Assessment to Practice A Strategy

• Assessment Instruments
• Defining the goals of an assessment project
• Designing assessment instruments
• Identifying the various key stakeholders
• Reporting Strategies
• Comprehending the range of reporting methods
• Linking effective reporting methods and key
  institutional stakeholders
• Discussing how effective reporting can promote
  changes in institutional practice
• Examples in Action
• MAP-Works Student Reporting
• MAP-Works Faculty/Staff Reporting

4
Assessment Instruments
5
Assessment Instruments
Step 1 Understanding the Outcomes of the
Assessment
Step 2 Breadth or Depth of Assessment
Keys to Successful Assessment Instruments
Step 4 Appropriate Survey Items
Step 3 Use of Assessment Information
6
Keys to a Successful Assessment Instrument

• Step 1 What are the outcomes of the assessment?
• Program improvement?
• Measuring the climate?
• Measuring student learning?
• Support accreditation / program review?

7
Keys to a Successful Assessment Instrument

• Step 2 What is the breadth or depth of your
  assessment?
• To thoroughly educate your audience on a range of
  items (diagnostic assessment)?
• Strengths and weaknesses
• Participant needs
• Areas of improvement
• Longitudinal trends
• Implications and suggestions
• To quickly provide just the facts (targeted to
  a specific topic)?
• Summaries
• Focus on Outcomes

8
Keys to a Successful Assessment Instrument

• Step 3 What will your assessment inform?
• Internal to Institution
• Decision making
• Program evaluation and improvement
• Budget allocations
• Marketing and education
• Interventions
• Program development
• External to Institution
• Accreditation reports
• Grant reports
• Benchmarking
• Professional development
• Publicity for alumni news, local media, etc.

9
Keys to a Successful Assessment Instrument

• Step 4 What questions are on your instrument?
• Potential Problem Survey is long and confusing
• Solution Create survey that is focused on
• Improvement, not marketing
• Performance, not activity (measure outputs, not
  activity)
• But, most important, MISSION
• Potential Problem Survey design is bad factors
  have low reliability results display poor
  validity
• Solution Have survey experts design survey or
  involve people closely connected to issue help
  design survey
• Potential Problem Dont know where to begin
  improvement once analysis is finished
• Solution Must have both analyses/systems that
  help guide action planning

10
Reporting Strategies
11
Reporting Strategies
Step 1 Understanding the Audience
Step 2 Appropriate Form for Information
Step 4 Choosing the Best Delivery Method
Keys to Successful Reporting
Step 3 Using Valuable Information
12
Understanding the Audience

• Step 1 Who will be reading your assessment
  report?
• Internal Stakeholders
• Administrative decision-makers
• Boards of trustees
• Budget administrators
• Faculty or staff
• Internal governing bodies (Faculty/Staff Senate,
  Unions, etc.)
• Students
• External Stakeholders
• Accreditation bodies reviewers
• Alumni
• Community members
• Donors
• Grant reviewers
• Prospective students parents
• State and federal governments

13
Understanding the Audience

• Step 1 (cnt) What is your audiences experience
  with information?
• Quantitative vs. Qualitative
• How comfortable are they with statistics?
• Do they prefer narratives or numbers?
• Interest and experience - How much explanation
  regarding the
• Topic?
• Assessment methods?
• Results?
• Implications?
• Time available - How much time can or will they
  spend reading the results?
• Level - Will they use university level data?
  College level? Department level? Individual
  level?

14
Appropriate Form for Information

• Step 2 What is the best form for the
  information?
• Types of reports
• Executive summaries
• Comprehensive reports
• Assessment summaries
• Notes, brochures, flyers, and memos
• Institutional snapshots
• Interactive data
• Easily read?
• What does the report look like (size of font,
  appearance, visuals, etc.)?
• Do they want to read this? Does it draw them in?
  Does it intimidate them or overwhelm them?

15
Using Valuable Information

• Step 3 What is the content of the report?
• Importance Does this report
• Include important issues?
• Highlight and emphasize the important results?
• Differentiate between important and non-important
  results?
• Usefulness Does this report
• Discuss the implications of the results?
• Clearly link the results to practice?
• Help practitioners determine what should be done?
• Differentiate between useful results and
  interesting results?

16
Choosing the Best Delivery Method

• Step 4 How will you deliver the assessment
  results?
• Media formats
• Paper
• Electronic (websites, downloadable files, CDs,
  emails, etc.)
• Oral presentations
• Combinations
• Easily accessible?
• How hard is the information for Stakeholders to
  access or find?
• Can they find what they need quickly?

17
Examples in Action MAP-Works Student Report

EBIs best practices model of reporting
18
Who is Responsible?

• Who is responsible for student success on your
  campus?
• Enrollment Management/ Retention?
• Student Affairs?
• Academic Affairs?
• What information do you know about this
  first-year student?

Enrollment Management/Retention
Student Affairs
Academic Affairs
Student ID YD252952HS GPA 3.93SAT Verbal
29Location In stateGender FemaleRace
African AmericanAge 18Major Undecided
Do you really know them?
19
Paradigm Shift
Im struggling in my math class

• What would happen if
• ALL faculty/staff were responsible for student
  success?
• YOU knew student was struggling?
• Could you do something about it before it was too
  late?

Im thinking about transferring
Student Affairs
Academic Affairs
Residence Hall Staff
Im really homesick
I dont think I can afford college
My roommate and I argue all the time
20
MAP-Works Mission 4 Way
Academic Success Improve students' ability to
succeed academically by realigning behavior with
grade expectations and focusing on elements of
academic success
Retention Minimize percentage of capable
students who drop out due to issues that could
have been addressed by self-awareness or timely
intervention by staff/faculty
Student Development Facilitate the
establishment of relationships, address
homesickness, identify residence hall living
issues
Student Involvement Connect students with
campus resources to facilitate involvement with
student organizations and campus programming
21
MAP-Works History

• In 1988, Ball State had a number of concerns
• First-year students arrived with unrealistic
  expectations (academics, grades, housing, etc.)
• Retention rates were not as high as they wanted
  them to be
• Faculty and staff were concerned that identifying
  student issues at mid-term was too late
• Faculty and staff wanted better data about
  incoming students

Fall 2008 -approximately 40 campuses will use
MAP-Works!
2005, Ball State partnered with EBI to create
MAP-Works
1989 to 2004, Ball State used MAP in-house
1988, Ball State developed concept
22
MAP-Works Process
23
Survey and Profile Items
Understanding the Students Experience

• Academic Integration
• Academic Self-Efficacy
• Core Academic Behaviors
• Advanced Academic Behaviors
• Commitment to Higher Education

• Profile Information
• Student information like gender and
  race/ethnicity
• Entrance exam scores
• credit hours enrolled

• Self-Assessment
• Communication Skills
• Analytical Skills
• Personal Management
• Time Management
• Health and Wellness
• Potential Issues (stress, financial, etc.)

• Social Integration
• Campus Relationships
• Living Environment (on/off campus)
• Roommate Relationships
• Homesickness

24
Who Benefits from MAP-Works?

• Who benefits from MAP-Works?
• First-Year Students
• Departments like
• Housing Residence Life
• Academic Advising
• Enrollment Management / Retention
• First-Year Seminar Instructors
• Minority Student Affairs
• Athletic Department
• Student Activities
• Academic Assessment Institutional Research
• Upper Administration

25
MAP-Works Student Reports

• Student Reporting
• Individualized on-line report provided directly
  to students within days of assessment
• 3 Main Reporting Purposes
• Purpose 1 Realign expectations
• Purpose 2 Information to help them plan for
  their success
• Purpose 3 Connect with appropriate campus
  resources

26
Purpose 1 Realign Expectations (Example 1)
Report provides benchmark information to help
students realign expectations
REPORTING TIP Only provide the most valuable
information in an summary report. Keep the
interesting information for the larger report.
27
Purpose 1 Realign Expectations (Example 2)
Report provides benchmark information to help
students realign expectations
REPORTING TIP Know your audience! Consider using
very simple charts to relay complex information
(most students dont have experience reading
complex charts/tables).
28
Purpose 2 Plan for Success
REPORTING TIP Consider providing written
explanations to help the reader draw conclusions.
Report provides feedback to help student
understand need to plan for their future success
29
Purpose 3 Connect with Resources
School lists campus resources/offices that link
with reporting area
REPORTING TIP Consider providing additional
information the reader may want to access.
30
Evaluating MAP-Works Student Reporting
Step 1 Understanding the Audience Students
have less experience with information reporting
is easy to read
Step 2 Appropriate Form for Information
Extremely visual and colorful. Minimal
information per slide.
Step 4 Choosing the Best Delivery Method Web
based is a delivery method most expected by this
generation
Keys to Successful Reporting
Step 3 Using Valuable Information - Targeted to
helping them be successful in school
31
Examples in Action MAP-Works Faculty/Staff
Report

32
MAP-Works Faculty/Staff Reporting

• Interactive system for faculty and staff
• Residence hall staff
• Academic advisors
• First-Year Seminar Instructors
• Other staff positions?
• 3 Main Reporting Purposes
• Purpose 1 Identify students who may benefit from
  personalized attention
• Purpose 2 Provide information for one-on-one
  meetings with students
• Purpose 3 Provide input regarding programming
  and training needs

33
Purpose 1 Identifying Students (Example 1)
REPORTING TIP Consider creating high level
dashboards that provides a lot of information
in an easy-to-read format.
These students are not adjusting academically or
socially and are not committed to their
institution.
34
Purpose 1 Identifying Students (Example 2)
REPORTING TIP Consider using simple color codes
instead of statistical information to relay
information.
These students very interested in leadership
positions Recruit them!
These students are moderately interested in a
leadership position Convince them?
These students definitely not interested in a
leadership position Why? Important on résumé
35
Purpose 2 Individual Meetings (Example 1)
REPORTING TIP Consider creating dashboard
reports that give a 30,000 foot look at the data.
Kimberly is having issues adjusting to college
(both academically and socially) and early
warning indicators are not good.
36
Purpose 2 Individual Meetings (Example 2)
REPORTING TIP Consider providing easy access to
in-depth information if the reader needs more
clarification.
37
Purpose 3 Programming
REPORTING TIP Consider that the information you
provide can be used in multiple ways and how that
information can be presented to best tell the
story.
These students have self rated themselves low in
Public Speaking skills. How could this be
addressed through programming?
38
Evaluating MAP-Works Faculty/Staff Reporting
Step 1 Understanding the Audience Reporting is
color-coded for readability No special training
required
Step 4 Choosing the Best Delivery Method Web
based is a delivery method that provides most
flexibility in searching through information
Step 2 Appropriate Form for Information
Extremely visual and colorful. Easy navigation.
Keys to Successful Reporting
Step 3 Using Valuable Information Highlights
students with more critical issues dashboards
are focused on critical transition issues
39
Additional Questions and Discussion
Darlena Jones, Ph.D. Director of Research and
DevelopmentEducational Benchmarking
Inc Darlena_at_webebi.com
For more information about MAP-Works,
contactTodd Pica, Todd_at_webebi.com Or, visit
www.MAP-Works.com