Assessment Toolbox for International Educators

1
Assessment Toolbox for International Educators

• Presenters
• David Comp, Darla K. Deardorff,
• Elaine Meyer-Lee,
• Lee Sternberg, Victor Savicki

2
Session Overview

• Assessment Overview and Context
• Highlights of Specific Assessment Tools/Methods
• Tools/Methods Exploration through Roundtable
  discussions

3
Assessing International Education An Overview

• Dr. Darla K. Deardorff
• Duke University
• d.deardorff_at_duke.edu

4
Why assess international education?

5
Why assess international education?

• Quality improvement
• Advocacy
• Accreditation
• Student learning

6
Why Assess? Measure success

• Moving beyond
• OUTPUTS to OUTCOMES
• What are meaningful outcomes (results) of
  internationalization efforts?

7
__________________________________________________
________________________   INTERNATIONALIZATION at
institutions of higher education   Inputs/Resourc
es needed for implementation of components of
internationalization Activities/Components of
Internationalization (college leadership, faculty
involvement, curriculum, study abroad,
international students/scholars/faculty,
international co-curricular units) Outputs of
Internationalization (i.e., number of
international students, number of study abroad
programs, number of students studying foreign
languages, etc.) Outcomes of Internationalizatio
n Intercultural competence what is it? How do
higher education administrators define it?
intercultural experts? How can it be
assessed?   Long-Term Impact of
Internationalization _____________________________
__________________________________ Figure 2.
General program logic model applied to
internationalization. (Deardorff, 2004, Page 58)
8
Ways of approaching assessment

• Some key points and
• lessons learned

9
Assessment-Contexts

• At Home curriculum, extra-curricular
  activities, community involvement/impact,
  domestic/internatl student interaction, policy,
  financial, campus climate
• Abroad (Cross-border) involves cross-border
  delivery of education through exchange, distance
  and e-learning, branch campuses, partnerships,
  host community impact

10
Assessment Lessons Learned

• Collaborate - Put together Assessment Team
• Adapt - Build on what you have
• Measure what is valued (align!)
• Plan - Develop assessment plan
• Use use what you assess
• Support from leadership, stakeholders
• Deardorff, 2008

11
ASSESSING OUTCOMES

• Starting point
• Mission Statement
• into
• Goals
• into
• Measurable Objectives

12
Assessment Cycle

• Define outcomes (based on mission/goals) and
  establish measurable criteria
• Identify appropriate assessment methods
• Collect data
• Analyze data
• Reflect on needed changes
• Design and apply changes

13
Assessment Lessons Learned

• Multi-method, multi-perspective
• Ongoing
• Integrated

14
Student Involvement in Assessment

• Student outcome assessment is not a measurement
  of learning but an integral part of learning. Can
  include language/disciplinary assessment,
  personal growth, intercultural competence, etc.
• Involve students in negotiating outcomes

15
Assessment Methods

• Methods determined by clearly articulated
  assessment questions what do we want to
  measure?
• Include
• DIRECT METHODS authentic assessment,
  portfolios, observation
• INDIRECT METHODS surveys, focus groups,
  interviews
• COMBINATION OF QUANTITATIVE AND QUALITATIVE

16
DIRECT METHODS

• Embedded course assessment
• Portfolios
• Performance
• Testing
• Papers/projects
• Capstone

17
Indirect Methods

• Surveys (inc. self-assessments)
• Interviews
• Focus groups
• Curriculum/transcript analysis
• Documented data

18
ICC Assessment Tools (85!)

• Intercultural Development Inventory
• Cross Cultural Adaptability Inventory
• Intercultural Conflict Styles Inventory
• Language Strategies Survey
• Strategies Inventory for Culture Learning
• Beliefs, Events, Values Inventory
• GAP Test
• Assessment of Intercultural Competence

19
Examples

• Georgia Techs International Plan uses
  capstone, IDI
• Dukes teacher ed program uses self-report
  instruments, embedded assessment,
  faculty/supervisor observation, focus group
• BCA uses embedded assessment, lang. tests, BEVI,
  resident director and host family observations
• ACE uses e-portfolio and self-report form

20
Questions to consider

• What are the specific goals and objectives?
• What does the tool/method specifically measure?
• Limitations and cultural biases of tools/methods?

21
More questions to consider

• How will assessment data be utilized? How will
  data be used to provide feedback to students? To
  improve the program/curriculum, etc?
• Is assessment multi-method, multi-perspective,
  ongoing, intentional, integrated, part of
  strategic plan?

22
OVERVIEW OF ASSESSMENT TOOLS

• GPI, American Identity,
• SCAS, BEVI

23
Assessment Toolbox for International Educators
GPI

• David Comp
• Senior Adviser for International Initiatives in
  The College
• The University of Chicago
• dcomp_at_uchicago.edu
• NAFSA Association of International Educators
• Annual Conference
• Washington, D.C. May 30. 2008

24
Global Perspectives Inventory (GPI)

• Developed by Larry Braskamp, David Braskamp
  Kelly Carter Merrill.
• -Larry Braskamp is a senior fellow at the
  Association of American Colleges and Universities
  (AACU). He is professor emeritus at the School
  of Education at Loyola University Chicago where
  he also served as senior vice president for
  academic affairs for four years. Braskamp also
  served as the first executive director of the
  Council for Higher Education Accreditation (CHEA).

25
Global Perspectives Inventory (GPI)

• The GPI is a survey of 46 items specifically
  designed to provide self-reports of students
  perspectives in three domains of holistic student
  development--cognitive, intrapersonal, and
  interpersonal.
• The GPI can provide evidence of students global
  perspectives
• At entry, during and at graduation from college
• At entry and conclusion of an education/study
  abroad experience
• Braskamp, Braskamp Carter Merrill, 2008

26
Global Perspectives Inventory (GPI)

• Theoretical Background of GPI
• The construction of GPI is based on the
  developmental perspective that students are on a
  journey during college. In this journey,
  students are given opportunities to reflect on
  three big questions
• How do I know? reflects the Cognitive
  dimension.
• Who am I? reflects the Intrapersonal dimension.
  
• How do I relate to others? reflects the
  Interpersonal dimension.
• Braskamp, Braskamp Carter Merrill, 2007

27
Global Perspectives Inventory (GPI)

• Description of the six GPI scales
• The Cognitive domain consists of two scales
  Knowing and Knowledge.
• The Intrapersonal domain consists of two scales
  Identity and Affect.
• The Interpersonal domain consists of two scales
  Social interactions and Social responsibility.

28
Global Perspectives Inventory (GPI)

• Validity and Reliability
• Reliability. Four GPI scales each loaded on their
  own independent factors (some with the exception
  of one or two items) considering a combination of
  both the highest and second highest loading
  scores Cognitive/Knowing, Cognitive/Knowledge,
  Intrapersonal/Affect, and Interpersonal/Social
  Responsibility. The Intrapersonal/Identity and
  Interpersonal/Social Interaction scales did not
  seem to load onto any independent factor. The
  overlapping of scales on each of the factors is
  not troubling, but rather encouraging.
  Theoretically, scholars agree these factors would
  all contribute to one concept global
  perspective.
• Validity. As students progress through college
  their development is likely to increase (Baxter
  Magolda, 1992 Pascarella Terenzini, 2005).
  Presumably, therefore, as people get older their
  global perspective may increase as well.
• Braskamp, Braskamp Carter Merrill, 2007

29
Advantages and Disadvantages

• Advantages
• GPI can be administered at
• entry, during and at graduation from college
• entry and conclusion of an education/study
  abroad experience
• GPI has only 46 survey questions and brief
  demographic questions
• GPI was developed by a leading expert in higher
  education assessment
• Disadvantages
• GPI is a relatively new assessment tool
• GPI has not been used in any research studies

30
New Tools for Intercultural Learning Outcomes
Assessment

• Elaine Meyer-Lee
• Director
• Joy Evans
• Assistant Director for Research and Scholarship
• Saint Marys College
• Center for Womens InterCultural Leadership
• Notre Dame, Indiana

31
For five years we conducted a comprehensive study
of our intercultural and international learning
on and off campus. For study abroad we used a
multi-method longitudinal assessment of
intercultural engagement, IC sensitivity,
identity, and goals
32
Evaluation Assessment Goals

• Evaluate programs effectiveness
• Measure Students
• Changes in sensitivity to cultural differences
• American identity development
• Own sense of growth toward goals
• Explore correlations of change with
• Intercultural engagement while abroad
  (interaction and reflection)
• Program or demographic characteristics

33
American Identity Measure

• Adaptation of Jean Phinneys Multigroup Ethnic
  Identity Measure (good track record), and her
  much newer American Identity Questionnaire
  developed for underrepresented groups
• Based on Eriksonian identity development theory.
  Measures 2 Factors
• identity search (e.g. I have often talked to
  other people about what it means to be an
  American.)
• affirmation, belonging, and commitment (e.g.
  Being American plays an important part in my
  life.)

34
American Identity Measure

• Adapted it (with Phinneys blessing) for study
  abroad students because one goal is more
  complexity in understanding ones own culture
• Students in returnee courses felt this national
  identity version captured an important dimension
• 10-item Likert-style Self Assessment (5 for each)
• Currently refining adaptations psychometric
  properties, but alpha coefficients good so far
  (.79)

35
Sources

• Phinney, J. (1992). The Multigroup Ethnic
  Identity Measure A new scale for use with
  adolescents and young adults from diverse groups.
  Journal of Adolescent Research, 7, 156-176.
• Phinney, J., Devich-Navarro, M. (1997).
  Variations in bicultural identification among
  African American and Mexican American
  adolescents. Journal of Research on Adolescence,
  7, 3-32.

36
Measure of Intercultural Engagement

• Asks students to quantify the frequency of their
  engagement through
• Interaction with cultural resources
• Interaction with local people
• Explicit reflection on these interactions

37
Benefits to Study Abroad Assessment
Challenges to Institutional Research

• MEI
• Comparable program specific data on students
  intercultural interactions
• AIQ
• Examine identity dimension, connect to literature
  on this

• MEI
• Labor-intensive development and administration
• AIQ
• Reliability and Validity of adaptation still
  being established
• Needs refining

38
Intercultural Leadership Certificate Program

• Capture Saint Marys strengths and CWILs unique
  location at the intersection of intercultural
  (local and global) and leadership education in
  the context of a womens college
• Guide students through a developmental process
  into a deeper capacity for intercultural
  leadership
• Bring coherence and provide an organizing
  framework for every effort (curricular and
  co-curricular) being undertaken at Saint Marys
  College to encourage intercultural leadership
• Portfolio assessment of students proficiency as
  an intercultural leader in
• 6 proficiency areas
• Recognize the Leader Within
• Articulate Your Ethical/Spiritual Center
• Engage With Value Diversity
• Dialogue on Power Privilege
• Create Inclusive Equitable Community
• Make Your Difference in the World

39

• For more handouts or bibliography on assessing
  intercultural outcomes of study abroad programs,
  go to
• http//www.saintmarys.edu/cwil/php/intercultural
  .learning/IILOutcomes.php
• or email me meyerlee_at_saintmarys.edu

40
Assessment Toolbox for International Educators
SCAS

• Victor Savicki
• Western Oregon University

41
Sociocultural Adaptation Scale (SCAS)

• Developed by Colleen Ward and Antony Kennedy
• Ward, C. Kennedy, A. (1999). The Measurement of
  Sociocultural Adaptation. International Journal
  of Intercultural Relations, 22, 659-677.
• Several subsequent supporting studies..

42
ABC Theory of Acculturation

• Ward, C. (2001). The A, B, Cs of acculturation.
  In D. Matsumoto (Ed.), Handbook of Culture and
  Psychology. (pp. 411-446). NY Oxford University
  Press.
• A Affect based on stress and coping theory
• B Behavior based on learning cultural knowledge
  and skills
• C Cognition based on social identification
  theory..

43
Sociocultural Adaptation Definition

• SCAS measures the Behavior aspect of the ABCs
• The ability to fit in, to acquire culturally
  appropriate skills and to negotiate interactive
  aspects of the host environment.
• Applied to both foreign nationals coming to the
  U.S and U.S. nationals going abroad..

44
SCAS Structure

• 29 items (reduced from an original pool of 41
  items)
• Making friends, Using the transport system, Going
  shopping, Dealing with unsatisfactory service,
  Understanding cultural differences, Understanding
  the locals worldview, Understanding the local
  value system
• Responses given on a scale rating the difficulty
  the respondent has in dealing with each item. 1
  (No difficulty) to 5 (Extreme difficulty)
• Takes about 5 minutes to complete
• Based-16 samples mostly from Australia, New
  Zealand, and Singapore..

45
SCAS Scales

• SCAS Total Sum/average of 29 items
• Two sub-scales based on a factor analytic study
• Cultural Empathy and Relatedness (13 items)
  (32.1 of variance)
• Understanding local perspectives, values, world
  views, and communication
• Impersonal Endeavors and Perils (7 items) (8.7
  of variance)
• Management of impersonal interactions and/or
  awkward situations..

46
Advantages and Disadvantages

• Based on a theory of acculturation
• Tracks key intercultural adaptation issues
• Gives several scores to interpret
• Predicts an overall progression of adaptation
• Can be used formatively and summatively
• List may be incomplete
• List may have cultural bias..

47

• The Beliefs, Events, and Values Inventory
• (BEVI)
• Developed by Craig N. Shealy, Ph.D.
• James Madison University
• shealycn_at_jmu.edu
• http//www.jmu.edu/ibavi

48

• The Beliefs, Events, and Values Inventory
• (BEVI)
• What does the BEVI measure?
• The BEVIis designed to assess a number of
  processes relevant to international experiences
  including (but not limited to) basic openness
  receptivity to different cultures, religions, and
  social practices the tendency (or not) to
  stereotype in particular ways self and emotional
  awareness and preferred but implicit strategies
  for making sense of why other people and
  cultures do what they do (from Shealy, 2005)

49

• The BEVI
• Has been in development since the 1990s
• Is derived from belief-value statements
• Is comprised of three validity and ten process
  scales
• Does not appraise correctness of individual
  items but an overall response pattern
• Is valid and reliable over time

50
The BEVI

• Has 54 demographic items including age, gender,
  ethnic background, educational attainment,
  political orientation, and religious affiliation
• Has 391 items across a wide set of beliefs and
  values rated on a four-point Likert scale
• Takes 45-55 minutes to complete
• Is available online

51
BEVI items include
I am never sad. I have felt jealousy toward
someone I loved. There is no such thing as
destiny. You can't argue with fate. We
should do more to help minority groups in our
culture. Big corporations often harm the
earth. I know that evil people go to hell when
they die. Hinduism and Buddhism have much to
teach the modern world.
52

• The BEVI
• Is effective in showing how peoples beliefs
  and values change as a result of having an
  international experience, including the way
  people feel about their own cultures, the value
  of learning about other cultures, the importance
  of learning a second language, the value in
  learning about world events, and other
  attitudes.
• Measures complex phenomena and is not
  face-valid
• Administration time can be long for some

53
(No Transcript)
54
Your Turn!

• Join us for Roundtable Discussions on the
  specific tools youve just heard about ask more
  questions, learn more in detail about how and
  when to use these instruments