Title: Stages of Concern During Curriculum Change: Formative Evaluation
1Stages of Concern During Curriculum Change
Formative Evaluation
- India Broyles EdD and Mildred Savidge PhD
- University of New England,
- College of Osteopathic Medicine
- 11 Hills Beach Rd, Biddeford, ME 04005
2What is the Difference in Formative and
Summative Evaluation?
- "When the cook tastes the soup, thats
formative. - When the guests taste the soup, thats
summative."
http//www.designedtoat.com/food2.shtml
3Learning Outcomes
Workshop Objectives
- The workshop will allow participants to
- Value formative evaluation during change
- Complete a sample of the SoC questionnaire
- Adjust the questionnaire to meet their own
innovation or curriculum change - Score the questionnaire and to make preliminary
analyses of different types of profiles - Predict ways in which SoC data can facilitate the
change process
4Agenda
- 10 minutes Formative Evaluation of the change
process - 20 minutes Complete the SoC questionnaire with
competency-based education as the innovation.
Score and report. - 15 minutes Understanding the stages of concern
and the SoCQ - 15 minutes What innovations are you
implementing? Rewrite the questions. Exchange
with a partner to review/edit the items.
Questions? - 20 minutes Analysis of the data
- 10 minutes What does the college do with the
formative evaluation data?
5Formative Evaluation
Formative Evaluation
- Formative evaluation can serve as a guide for a
college in the change process and improves the
chances of a successful curriculum innovation
that will be both well accepted and long lasting.
6Formative Evaluation
- Formative evaluation is a method of judging the
worth of a program while the program activities
are forming or happening. - Formative evaluation focuses on the process.
- (Bhola 1990).
7Formative Evaluation typically involves
- gathering information during the early stages of
your project or program - a focus on finding out whether your efforts are
unfolding as planned - uncovering any obstacles, barriers or unexpected
opportunities that may have emerged - identifying mid-course adjustments and
corrections which can help insure the success of
your work (NW Regional Education Lab) - How do we do this?
8The process of developing and implementing an
innovation is
- Change
- So we look at the literature on educational
change.
9Individual Innovativeness TheoryCarl Rogers
(1995)
- Innovators
- Early Adopters
- Early Majority
- Late Majority
- Laggards
10Change is an ongoing process, not a short-term
event.
- Change requires ongoing support and resources
and it takes time. Sometimes new users of an
innovation get discouraged when they don't see
immediate results. It is important to have
realistic expectations about the time it will
take to see significant progress and to make sure
that other stakeholders in the community
understand this as well. Failure to address key
aspects of the change process can either add
years to, or even prevent, successful
implementation. - Gene Hall, Shirley Hord, Archie George.
Southwest Educational Development Lab
11Change occurs in individuals first, then in
organizations.
- The best curriculum in the world will not succeed
in your institution unless the people
implementing it are ready and willing to make it
a success. However, individual change is
difficult if the organization is not supportive
of the change. - Gene Hall, Shirley Hord, Archie George.
Southwest Educational Development Lab
12One Change in Medical Schools
- Competency-based Education with emphasis on
national core competencies. -
- Please complete the questionnaire. Takes about
10 minutes. Score using quick-scoring guide.
Create your profile. You will have 20 minutes
total. Give the quick scoring guide to Millie to
record. Be sure your name is on it.
13Agenda
- 10 minutes Formative Evaluation of the change
process - 20 minutes Complete the SoC questionnaire with
competency-based education as the innovation.
Score and report. - 15 minutes Understanding the stages of concern
and the SoCQ - 15 minutes What innovations are you
implementing? Rewrite the questions. Exchange
with a partner to review/edit the items.
Questions - 20 minutes Analysis of the data
- 10 minutes What does the college do with the
formative evaluation data?
14What are Concerns?
- The world around us is complex. It is not
humanly possible to focus at one one time on all
of the many different issues. Certain aspects of
our world are of higher priority. - The composite representation of the feelings,
preoccupation, thought and consideration to a
particular issue or task is called a concern.
15Concerns
- Depending on ones closeness to and involvement
with an innovation, ones concerns will be
different in type as well as in intensity. - Many types or levels of concerns can be
experienced concurrently. - There is a predictable pattern to the movement of
intensity of concern across types.
16Based on studies at UTEXAS, Austin since 1974
- Stages have been identified with a developmental
movement through these stages. - Earlier concerns must be resolved before later
concerns emerge. - Timely provision of affective experiences and
cognitive resources can facilitate resolution of
early concerns and development of higher
concerns. - Personalized interventions are important.
17Stages of Concern
- Stage 0 -- Awareness. Users have little concern
or involvement with the innovation. - Stage 1 -- Informational. Users have a general
interest in the innovation and -
would like to know more about it. - Stage 2 -- Personal. Users want to learn
about the personal ramifications of the - innovation.
They question how the innovation will affect
them. - Stage 3 -- Management. Users learn the
processes and tasks of the innovation. -
They focus on information and resources. - Stage 4 -- Consequence. Users focus
on the innovation's impact on learners. - Stage 5 -- Collaboration. Users
cooperate with other users in implementing the -
innovation. - Stage 6 -- Refocusing. Users
consider the benefits of the innovation and -
think of additional alternatives that might work
even better.
18Concerns-Based Adoption ModelGene Hall, Shirley
Hord, Archie George. Southwest Educational
Development Lab
- Stages of Concern
- Questionnaire
- Interviews
- Levels of Use
- Interviews
- Innovation Configuration
19SoCQ Instrument
- Reliability
- Coefficients of internal reliability
- Test/Retest correlations
- Validity
- Statistical
- Content
- Face
20Coefficients of Internal ReliabilityN 830
21Multiple studies Coefficients of Internal
Reliability for each stage
22Test-Retest Correlations N132
23Validity Studies Statistical
- Factor analysis
- 7 stages explained 60 of variance
- All had Eigen values above 1.0
- Hypothesized scales matched factor scales
- Each scale measured an independent construct
- Correlations between scale scores shows strong
positive relationship between contiguous stages
24Factor Analysis
25 Correlations between Scale Scores
26Validity Comparison Studies
- Focus on accuracy in measuring highest and lowest
concerns - Interviews and open-ended responses used
- Statements matched to scale scores/ profiles
- Qualitative analysis showing greatest and least
concerns correlated with scale scores - External judges used to establish relationships
between interview analysis and scores
27Validity Longitudinal Studies
- Two 2-year longitudinal studies
- Showed concerns changing over time in accordance
with concerns theory - Concerns measured by the instrument moved from
high Awareness, Informational and Personal
concerns to lower concerns on all levels, or
higher Consequence, Collaboration and Refocusing
concerns
28Use in clinical education
- Lewis and Watson (1997) measured the concerns of
57 nursing faculty about the use of computer
technology. Their pre-post study results suggest
that the primary concerns of the faculty were
informational and that addressing these concerns
through workshops increased interest in the
innovation. - Gwele (1996) measured the concerns of nurse
educators (n93) at four nursing colleges during
the implementation of a major legislated
curriculum reform. Concluded that when staff is
required to adopt a major curriculum change the
normal progression through the stages of concern
is impeded. They suggested that in these
situations it may be important to delay adoption
until participants can come to terms with the
need to adopt the new curriculum. - Arwer, Harris Dusold (2004) assessed the
concerns of staff during the installation of a
telemedicine system and to assure that concerns
were addressed during system implementation.
Survey findings were used successfully to modify
the implementation and training phases of the
program to better meet the needs of the staff.
29Components
- Cover Letter / Introductory Page
- 35 items
- Demographics
- Create a SoCQ for an innovation in your own
institution. (15 minutes)
30Interpretation of SoC Data
- Look at your scores/profile on CBE
- Peak Stage Score
- 1st and 2nd High Stage Score
- Profile Interpretation
31Peak Stage Scores
32Group Data
- Tally the number of individuals that are high on
each stage. This gives a clear picture of the
range of peak stage scores within a group (show
of hands - how many at each stage). - Aggregate individual data by developing a profile
that presents the mean scores of each stage of
the individuals.
33 Typical Non-User
34Institutional Profile of Stages of Concern
35Profile Interpretation
- As individuals move from unawareness and nonuse
of an innovation into beginning use and more
highly sophisticated use, it is hypothesized that
their concerns develop from early to late
concerns - Use clinical interpretation techniques
- The total score is not generally used in analyses
because it does not have a unique meaning.
36Institutional Profile, 2006
37Negative One Two Split
38Negative One -Two Split with Tailing Up at Stage
6
39Stages of Concern
- Stage 0 -- Awareness. Users have little concern
or involvement with the innovation. - Stage 1 -- Informational. Users have a general
interest in the innovation and -
would like to know more about it. - Stage 2 -- Personal. Users want to learn
about the personal ramifications of the - innovation.
They question how the innovation will affect
them. - Stage 3 -- Management. Users learn the
processes and tasks of the innovation. -
They focus on information and resources. - Stage 4 -- Consequence. Users focus
on the innovation's impact on learners. - Stage 5 -- Collaboration. Users
cooperate with other users in implementing the -
innovation. - Stage 6 -- Refocusing. Users
consider the benefits of the innovation and -
think of additional alternatives that might work
even better.
40Intense Management Concerns
41Consequence Concerns
42High Collaboration and Consequence Concerns
43Single High Collaboration Concerns
44Stages of Concern
- Stage 0 -- Awareness. Users have little concern
or involvement with the innovation. - Stage 1 -- Informational. Users have a general
interest in the innovation and -
would like to know more about it. - Stage 2 -- Personal. Users want to learn
about the personal ramifications of the - innovation.
They question how the innovation will affect
them. - Stage 3 -- Management. Users learn the
processes and tasks of the innovation. -
They focus on information and resources. - Stage 4 -- Consequence. Users focus
on the innovation's impact on learners. - Stage 5 -- Collaboration. Users
cooperate with other users in implementing the -
innovation. - Stage 6 -- Refocusing. Users
consider the benefits of the innovation and -
think of additional alternatives that might work
even better.
45High Refocusing Concerns
46High Management Concerns With Ideas
47Stages of Concern
- Stage 0 -- Awareness. Users have little concern
or involvement with the innovation. - Stage 1 -- Informational. Users have a general
interest in the innovation and -
would like to know more about it. - Stage 2 -- Personal. Users want to learn
about the personal ramifications of the - innovation.
They question how the innovation will affect
them. - Stage 3 -- Management. Users learn the
processes and tasks of the innovation. -
They focus on information and resources. - Stage 4 -- Consequence. Users focus
on the innovation's impact on learners. - Stage 5 -- Collaboration. Users
cooperate with other users in implementing the -
innovation. - Stage 6 -- Refocusing. Users
consider the benefits of the innovation and -
think of additional alternatives that might work
even better.
48Profile of Impact-Concerned User and Coordinator
49Unconcerned Innovation User
50Guidelines
- Establish a holistic perspective
- Look at High and Low scores
- Look at individual item responses
- When scores are used in statistical analyses,
developers recommend use of raw scores.
Conversion to percentiles greatly affects the
distribution of the scores.
51Demographics
- In previous research, no outstanding
relationships between standard demographics age,
gender, teaching experiences, etc. - What demographics would you choose?
-
52 Basic Scientist Profiles
- Responses from 8 basic scientists show high
concerns for awareness, but very different
concerns for other stages. All but two have
tailing up at Stage 6.
53Clinical Faculty Profiles
- Clinicians show more similar profiles at
Awareness, Information, and Personal Concerns.
Five clinicians have a high concern at
refocusing. Although unusual for non-users, we
are still in the development of several elements.
54Staff Profiles
- Similar to clinicians in early concerns, as
expected in non-users. Low management and
consequence concerns may reflect non-classroom
duties. The have strong concerns about
collaboration.
55Follow-up Interviews
56Themes from interviews
57What we learned
- The process of curriculum revisions needs to be
systematic with timelines and responsibilities. - Everyone needs to be kept informed of the process
and the outcomes of deliberations with
information coming in multiple formats - a website
- hallway bulletin boards
- written material
- key descriptive and research articles.
- Care should be taken to release information in a
logical fashion so that faculty and staff not
directly involved in the development stages can
understand the reasons for current plans and have
an opportunity to respond if they are unhappy
with plans at any given stage.
58Additional resources will be needed
- qualified assistance with effective use of WebCT
- grant-writing support
- faculty coaching
- implementation of effective evaluation approaches
59Understanding the stages of concern can result
- in more targeted strategies
- more relevant workshops
- directed planning to implement the new curriculum
plan thereby creating successful,
institutionalized change
60We believe
- The importance of setting up a curriculum
revision process based on both individual and
institutional concerns has the potential for
influencing the development and implementation of
a new curriculum.
61- Lewis, D. and Watson, J.E. Nursing faculty
concerns regarding the adoption of computer
technology. Computers in Nursing. 1997
15(2)71-6. - Gwele, N.S. Concerns of nurse educators regarding
the implementation of a major curriculum reform.
Journal of Advanced Nursing. 1996 24607-14 - Arwer, J.M., Harris, K. and Dusold, J.M.
Application of the concerns-based adoption model
to the installation of telemedicine in a rural
Missouri nursing home. Journal for Nurses in
Staff Development. 2004 2042-9.