SoTL and Assessment

1
SoTL and Assessment

• Sandi Regan
• Hayat Weiss
• Tom Laughlin

2
Assessment

• Formative Assessment Tasting the punch, can
  make changes to final product
• Summative Assessment Final product, less
  flexibility

3
Assessment

• Purpose is to engender competence
• Norms
• Process is connected to learners world, frames
  of reference, and value.
• Demonstration of learning
• multiple ways to represent knowledge and skills
• allows for attainment of outcomes at different
  points in time.
• Self-assessment is essential to the overall
  assessment process
• (Wlodkowski Ginsberg, 1995)

4
How much evidence is enough?

• Tendency to ask too broad a question
• Need to define in ways that can be measured
• Problems determining adequate measurement
• Being intimidated by educational research
  emphasis
• SoTL involves creation of new genres
• Being explorers rather than settlers

5
Games of LearningPiaget Comes Alive
Sandi Regan
6
Questions

• How can students construct their own ways of
  learning?
• What methods can help students develop genuine
  knowledge that they can retain and apply to their
  work with children?

7
Howard Gardner stated

• Understanding is itself a complex process that
  is not well understood a sufficient grasp of
  concepts, principles or skills so that one can
  bring them to bear on new problems and
  situations, deciding in which ways ones present
  competencies can suffice and in which ways one
  may require new skills or knowledge.
• (Wiggins, 1998)

8
One Method Doesnt Fit All

• Active Learning
• Multiple Intelligences
• Model Teaching
• Constructivism

9
Learning is constructing meaning.
constructivism maintains that students are
active agents in the learning process, exploring,
testing, and probing in the pursuit of knowledge,
understanding, and ultimately, personal meaning.
The learner is curious, independent, and
diligent Chickering and Gamson (2002)
10
Early Childhood Theory Puzzle or Game Assignment

• create a visual game for other students that
  focuses on the concepts and/or stages of an
  particular early childhood development theory or
  many theories
• Piagets Cognitive Development
• Eriksons Psychosocial Development
• Maslows Hierarchy of Needs
• Behaviorism
• Bronfenbrenners Ecological Systems, etc.

11
Goals for Students for Game/Puzzle Assignment

• Learn about theories by creating game
• Provide information and help other students to
  learn and genuinely understand the theories by
  playing each games
• Explore multiple views of theories by playing
  other students games

12
Assessments

• Theory game or puzzle checklist
• Verbal feedback
• Pretests and posttests
• Second posttest
• Students assessments

13
(No Transcript)
14
Students Assessments of Creating and Playing
Each Others Games

• Verbal Feedback
• Ongoing Student Remarks
• Survey

15
Question 1
How helpful was creating your theory game in
helping you understand that particular theory?
16
Students Indications (43 people answered
survey.)

• Scale
• ___________________________________
• 1 2 3 4 5 6 7
  8 9 10
• no effect helpful very helpful
• of people
• 1 2 14 1 2 3
  2 18

17

• Question 2
• How helpful was doing the game activities in
  Early Childhood Growth Development class in
  helping you understand ECE theories?

18
Students Indications (43 people answered
survey.)

• Scale

___________________________________ 1
2 3 4 5 6 7
8 9 10 no effect
helpful very helpful of people
1 14
3 4 21
19
Samples of Students Quotes

• Key information given helped bring the learned
  info to the front of my mindI definitely
  recommend developing questions about a theory and
  setting up flashcards to use with the answers on
  back. I learn best if I am teaching someone
  else.
• Sometimes I get confused on different theories
  and the stages, but by doing this I now can
  recognize Freuds stages. Its a lot easier to
  learn when you are enjoying yourself.
• Because unlike reading information, I was forced
  to present information which caused me to
  actually use it while designing my poster. After,
  it just became known to me as knowledge.

20
Added Benefits Playing The Games

• Student initiated conversations about the
  theories
• longer and deeper discussions about the theories
• OPPORTUNITIES
• to address misunderstandings students had of
  theory concepts
• to discuss the errors in some games or students
  answers that were incorrect in a laidback,
  relaxing atmosphere

21
Getting more mileage

• How can you connect your assessment approach with
  other initiatives at your college or in your
  discipline?

22
Value of SoTL Approach

• Informs teaching in ongoing way
• Makes classroom more interesting and relevant for
  everybody
• Share with other colleagues keep ideas vibrant
• Recognize value of keeping student evidence over
  time
• Decreases isolation and boredom

23
Scholarship of Teaching and LearningIn a
Mathematics ClassroomAlgebra I

• Hayat Weiss
• Middlesex Community College

24
Student Placement Results

• Placed into Developmental English 55
• Placed into Developmental Reading 29
• Placed into Developmental Math 82
• Honors 10

25
Algebra I Background Context

• Longstanding problem of course completion in my
  Algebra classes from 1986 until 1999
• Course completionPassing course with D- or
  better
• I did not differentiate between F and W grades
• Completion was slightly over 50 and varied from
  class to class anywhere between 27 and 77

26
Interventions to Improve Course Completion in
Algebra

• Interventions prior to 1999
• Called students after they missed 3 classes
• Sent warning letters
• Met with students individually in my office
• In 1999, I joined the Collaborative Intervention
  Teams project.
• In Fall 02, I joined the Carnegie Community of
  Practice

27
The Collaborative Intervention Teams Model

• Number of Teams across the college per semester
  4-10

28
Team Participants

• Faculty Members from different disciplines
  Math, English, Technology,..
• LD Specialists
• Academic and Career Counselors
• Health Services Staff
• Academic Support staff

29
The Way Teams Functioned

• Teams consisted of a faculty member and one
  other person.
• Teams were autonomous. Each team worked out its
  own plan of action.
• All teams met as a group once or twice during the
  semester to exchange ideas and report on their
  activities.

30
Algebra Team ModelFaculty Learning Specialist

• Learning specialist responsibilities
• Attend every class except test days
• Administer learning tools during second week of
  semester
• -Study Habits Assessment Survey
• -Learning Styles Inventory
• Help students individually.

31
Student Tutor (only 2 semesters)

• Student tutor responsibilities
• Attend class once a week
• Check homework and return to students during
  same class
• Interact with students as time allowed
• Our data showed that collecting homework made no
  difference in course completion. Needs
  revisiting.

32
Students response to having an assistant in
the classroom

• Survey results
• 11 said it made a strong difference
• 45 said it made somewhat of a difference
• 44 said it made no difference
• Two representative comments
• _If I did not remember most of it from HS
  Math, I might have been more interested. That is
  what I get for not reviewing for the placement
  test.
• _One might be able to explain things better
  for you.

33
Additional Interventions

• Feedback form
• Faculty decided to seek feedback regularly
  from students and devised a form that students
  were asked to fill every 4 weeks.
• Changed approach to teaching some topics,
  especially word problems (in Fall 02)

34
Feedback Form

• Students set realistic, concrete goals for
  themselves
• They measure their study time and keep track of
  their grades
• They practice making positive statements
• They look for occasions to reward themselves
• They go over the old forms to see if they
  accomplished their goals.
• Students tell us how we can help them

35
Benefits and Drawbacks of Feedback Form

• Benefits
• -Students stay involved in their own learning
• -The language emphasizes a positive attitude
• -Students evaluate their performance every few
  weeks, so no surprises at the end of the semester
• -Students own responsibility is continuously
  reinforced
• Drawbacks
• -Takes too much time and too much effort
• -Sometimes we cannot do what they ask for (ex.
  give less homework)
• -Boring and repetitive.

36
Outcomes and implications(from Administration)

• MA 1103 Algebra I - Spring 1999

37
Outcomes and implications(from Administration)

• MA 1101 Fundamentals of Math -Spring 1999

38
Outcomes and implications(from Administration)

• EN 1101 Basic Writing - Spring 1999

39
Questions

• Are these interventions making any difference in
  student learning?
• What is student learning? How do we measure it?
• What is the best way to measure outcomes in a
  systematic way?
• Do these efforts have any effect on course
  completion?

40
Analysis

• I compared completion rates between Regular
  classes and Collaborative classes starting in F99
• I compared completion rates in all classes prior
  to F99 and in all classes (the two types grouped
  together) from F99 on.

41
Comparative data
42
Statistical Values F99-S02
43
Regular Classes Vs Collaborative Fall 99-S02

• Regular classes box plot on top and collaborative
  below
• .56 .665 .75
• .47 .75
• .54 .64 .71
• .48 .78

Regular
Collaborative
44
NO Difference between the two types of classes!

• Completion regular classes pr .6393
• Completion collaborative classes pc .627
• It seems no difference but lets check
• Hypothesis test Ho pr pc
• Ha pr gt pc
• P-value .41
• Assuming Ho is true, we have a 41 chance of
  getting such results due to normal variation.
  Therefore, there is No significant difference.

45
What Happened?All that work for nothing?

• Did more students with learning disabilities sign
  up in the collaborative classes?
• Could it be that there is a crossover effect
  between the collaborative courses and the regular
  classes?
• In other words, did some of the methodology
  affect my other Algebra classes?

46
How do I check it out?

• The number of disabilities students did not seem
  to be a factor
• How do I check for crossover effect?
• I compared data prior to Fall 99 with data from
  Fall 99 on.

47
Is there a difference between F93-S99 and F99-S03
48
Is the Difference Between the Proportions
Significant?

• Completion F93-S99 p1171/ 337 .507
• Completion F99-S03 p2 184/ 291 .632
• Is the difference significant or is it due to
  chance and normal variation?
• Hypothesis test Ho p1 p2
• Ha p1 lt p2
• The test resulted in a P-value .0008
• Assuming that p1 p2 there are 8 chances in
  10,000 of getting such results. That is very
  rare therefore we conclude that there is a
  significant improvement in course completion.

49
Where is the Improvement Most Apparent? Box
plots F93-S99 F99-S03

• .44 .5 .62
• .27
  .78
• .54 .65
  .74
• .47
  .78

F93-S99
F99--S03
50
Conclusion

• It seems that there is a crossover effect
• The learning specialist became more comfortable
  with the course and was able to help all students
  outside of class, not just the students signed up
  in the collaborative course.
• Interventions did make a difference
• Maximum rate stayed the same. No Algebra class
  got better than 78 completion rate.
• The bottom 3 quarters were pushed up
  dramatically.
• The lowest completion rate went up from 27 to
  47.

51
Electronic Communication in a Basic Writing
Classroom What Do Students Gain?

• Tom Laughlin
• Middlesex Community College
• Bedford, MA

52
Other Basic Writing Instructors Experiences
Using Computers in Classrooms

• Computer-related assignments increase student
  involvement in their own educations
• Provides variety and adds interest
• Positively influenced students attitudes
• Students feel they are learning the technology of
  the future
• Contributes to the social needs of some students,
  builds community

53
Other Basic Writing Instructors Experiences
Using Computers in Classrooms

• More comfortable learning environment for
  nonverbal, shy, socially marginalized students
• Students are not limited to the improvisational,
  spontaneous style required in most classroom
  discussions
• Helps create a more student-centered classroom

54
Other Basic Writing Instructors Experiences
Using Computers in Classrooms

• Fosters collaborative learning
• Encourages writing for an audience other than the
  teacher
• Students develop confidence and learning skills
  that help them in other classroom environments

55
My Interests

• Exploring ways that synchronous and asynchronous
  communication might enhance my students written
  communication

56
My Hopes

• To increase students comfort with electronic
  communication
• To increase the fluency of students written,
  electronic communication
• To improve the quality of students written,
  electronic communication

57
Alexander W. Astin, The Civic Challenge of
Educating the Underprepared Student (2000)

• As educators we value being smart much more than
  we do developing smartness we forget that our
  institutions primary mission is to develop
  students personal capacities, not merely to
  select and certify those students whose
  intellectual talents are already well developed
  by the time they reach us.

58
Wlodkowski Ginsberg,Diversity and motivation
(1995)

• Self-assessment allows us to gain perspective on
  how we understand ourselves as learners, knowers,
  apprentices in a discipline, and citizens in a
  complex and paradoxical world

59
Cross Steadman, Classroom Research
Implementing theScholarship of Teaching (1996)

• Research on teaching and learning in classrooms
  cannot exclude those who are most involved and
  most affected, namely teachers and students.

60
Changes to my Basic Writing class

• Met 1 day / week in networked classroom held
  electronic discussions
• Developed electronic assignments
• Involving students in readings, discussions (oral
  and electronic), self-reflection, and written
  formal papers about electronic communication

61
Changes to my Basic Writing class

• A formal essay assignment asked students to
  explore the advantages and/or disadvantages of
  electronic communication in college by using own
  personal experiences in our class (and perhaps
  in other classes at MCC)

62
My Questions

• Will the use of written, electronic
  communication in my basic writing classes
• a) increase students comfort with written,
  electronic communication?
• b) increase the length of students written,
  electronic communication?
• c) improve the quality of students written,
  electronic communication?

63
Assessments

• Pre and Post Surveys - Comfort levels
  Experience levels
• Length of written electronic entries
• Quality of written electronic entries
• Student reflections

64
STUDENT SURVEY ON COMPUTERS ELECTRONIC
COMMUNICATIONComfort Levels

• 3) How comfortable are you using a computer?
• 7) How comfortable are you with meaningful
  electronic written
• communication and/or serious electronically
  written discussions?
• 8) If a college course you were considering
  expected some written
• electronic communication (such as electronic
  chat, discussion boards, etc.), how might this
  influence your signing up for that course?
• 9) If a job you were considering expected some
  written electronic
• communication (such as electronic meetings,
  emailed reports,
• electronic discussions with co-workers or
  supervisors, etc.), how might this influence your
  job application?

65
RESULTS   1)
Comparisons between pre and post surveys of
students regarding comfort levels  
Survey Chart 1

66
STUDENT SURVEY ON COMPUTERS ELECTRONIC
COMMUNICATIONExperience Levels

• 4) How much experience have you had using a
  computer for writing?
• 5) Have you used electronic communication (such
  as email, chat, etc.)?
• 6) How much experience have you had with
  electronic communication (any type of
  communication with others using computers)?

67
RESULTS   Comparisons between pre and post
surveys of students regarding experience levels  
Survey Chart 2  
68
STUDENT SURVEYOpen-Ended Questions Regarding
Benefits" and Disadvantages"

• 10) What do you see as some possible benefits for
  you and/or others when communicating
  electronically?
• 11) What do you see as some possible
  disadvantages for you and/or others when
  communicating electronically?

69
Students Pre-survey Responses to Open-Ended
Questions

• Benefits - rather general focused on
• a) improving typing and computer skills
• b) being more familiar with current technology

70
Students Post-survey Responses to Open-Ended
Questions

• Benefits - tended to be more specific
• and focused on
• a) more openness in expressing thoughts and
  feelings than with oral communication
• b) ease when using electronic communication

71
Students Pre- and Post-survey Responses to
Open-Ended Questions

• Disadvantages
• a) varying typing speeds
• b) limitations on individual access to technology
• c) use of electronic communication to the
  detriment of oral communication
• d) lack of verbal and physical cues
• e) impersonal nature of electronic communication

72
Comparisons of the length of students written,
electronic entries  
Word Count Chart - Males
Word Count Chart - Females
73
Basic Writing Exit Criteria

• 1) A relatively well developed and expressed main
  idea
• 2) A sense of introduction, conclusion, and
  organization
• 3) Most paragraphs developed around appropriate
  topic sentences
• 4) Sufficient relevant supporting details
• 5) Few if any fragments or run-ons that suggest
  lack of sentence sense
• 6) A reasonable grasp of rules for commas and
  apostrophes
• 7) Few serious spelling errors

74
Quality Comparison Chart samplePlease indicate
the degree of change between Sample 1A and Sample
1B for each of the criteria.

75
Comparisons of the quality of students written,
electronic entries
Writing Quality Chart
76
Basic Writing Exit Criteria

• 1) A relatively well developed and expressed main
  idea
• 2) A sense of introduction, conclusion, and
  organization
• 3) Most paragraphs developed around appropriate
  topic sentences
• 4) Sufficient relevant supporting details
• 5) Few if any fragments or run-ons that suggest
  lack of sentence sense
• 6) A reasonable grasp of rules for commas and
  apostrophes
• 7) Few serious spelling errors

77
Student quotes from formal papersSheila wrote

• Electronic communication, in my class, gives
  students an opportunity to express their feelings
  and opinions without any concerns. I have found
  that students respond more often when they are on
  the computer rather than sitting in class in
  front of other students. Maybe the computer
  gives them a sense of security, and allows them
  to speak more freely. I know that I am much more
  comfortable talking on a computer than in front
  of most people. Some people in my class are quiet
  and shy. When they get on the computer they seem
  to talk much more with other students.

78
Vicky wrote

• Another advantage to electronic communication I
  have experienced is class discussion using a
  discussion board. As I looked back at some of
  my responces I noticed some were long and very
  detaild. If I was to have this discussion in
  class face-to-face with other people I would not
  have been confident enough in myself, to say as
  much as I did in the discussion board.
  Electronic communication gives me the chance to
  say how I feel without having to say it to
  someones face.

79
Pete wrote (one of the less vocal students in
class)

• my computer voice makes it easier to discuss
  controversial issues. When I write someth ing on
  paper, I feel it can be saved and critisized
  for ever. This makes me nervous.

80
Phil wrote

• Electronic communication is a very important
  tool in college. Using the computer seems to
  make some of the pressure of just writing a paper
  go away. For example, in the Blackboard
  Discussion Board my classmates and I seem to be
  more open, and have more fun writing. I believe
  this helps to take some stress and anxiety away,
  and leads to better writing.Another big
  advantage of communication by computer is you
  get almost instant feedback from your classmates.
  This can be helpful in many ways. I use the
  feedback to look over what I wrote and see if I
  can improve on it. I hope to see more teachers
  and students taking advantage of this important
  tool we call the computer. I know it will help
  students communicate better with their teachers,
  and other students. When you get the chance to
  hear other peoples opinions and comments, you can
  teach yourself to be a better writer.

81
Denise wrote

• I dont think it is the same as our journals. We
  didnt write a complete response. We would say
  one thing and wait for response, then add to it
  later. It was more like we were talking about
  it, not writting about it.

82
Pete wrote

• The first thing I notice is I tend to write
  less on the computer. I think this can be very
  helpful because you get to the point quicker.
  In my opinion the topic can be made clearer by
  keeping it short and to the point.

83
Evan wrote

• Although there is much to praise on Electronic
  Communication there are some possible
  disadvantages. One being when people talk to
  each other online or type to each other if you
  will there is no structure no rules really. You
  can even drop vowels in words. Some people think
  that this hurts the people that are online
  particularly young people. They fear that the
  person will forget all rules that apply in
  English.

84
Denise wrote

• I noticed people using abreviations or acronyms
  when electronically chatting. When I first
  signed online, people were typing ASAP or
  TTYL. I did not know what was going on. I had
  to learn another jargon all over again, just like
  the first day of school. These abreviations and
  acronyms made it faster and easier to type, but
  typeing faster makes us forget correct
  punctuations and capital letters.
• In my opinion, I agree with electronic
  communication if you use it right. No one said,
  When you go online to chat, forget everything
  you learned in English growing up. It does help
  us keep in touch with distant friends and
  relatives, but we do not have to type like we
  never were taught to write.

85
My Questions

• Will the use of written, electronic
  communication in my basic writing classes
• a) increase students comfort with written,
  electronic communication?
• b) increase the length of students written,
  electronic communication?
• c) improve the quality of students written,
  electronic communication?

86
Providing Classroom Experiences with Synchronous
Electronic Communication

• Helped develop greater comfort positive
  experiences with technology meaningful
  electronic communication
• Helped students fluency in writing
• Contributed to an improvement in the quality of
  their written expression

87
Benefits of this SoTL work for me

• Reinforced my interest in the benefits of
  electronic written communication as part of
  developmental writing courses
• Thinking more about norms for electronic
  communication
• More conscious of gender differences in my
  classes
• Rethinking assessment measures

88
Electronic Communication in a Basic Writing
Classroom What Do Students Gain? Thomas S.
Laughlin Associate Professor of English,
Middlesex Community College In Explorations
from a Community of Practice The Scholarship of
Teaching and Learning at Middlesex Community
College, Spring 2004 Available through
www.middlesex.mass.edu/carnegie
89
  Explorations from a Community of Practice The
Scholarship of Teaching and Learning at Middlesex
Community College, Spring 2004
CONTENTS Introduction
90
Tutorial on Scholarship of Teaching and Learning

• from Indiana University, Bloomington
• http//webcenter1.aahe.org/sotl_tutorial/home.html

91
Framing the Question (Goal Approach)

• 1. Define a goal
• 2. Ask and answer questions to refine the goal
  (Colleagues are usually helpful.)
• 3. Create a succinct summary of a specified
  goal
• 4. Ask what evidence would reveal that the
  goal is achieved?
• 5. Frame possible researchable questions

92
Making Vague Questions Answerable Using
Operational Definitions

• Less framed
• Do students who help others learn an academic
  discipline learn it better themselves?
•  
• Do students learn more in small classes?

• More framed
• Do students in CMSC 250 who tutor students in
  CMSC 150 perform better on the CMSC 250 final
  exam than students who do not tutor but have
  similar grades in CMSC 150?
• Do students in sections of Phys 118 enrolling
  fewer than 50 students perform better on the
  departmental final exam than students from
  sections enrolling more than 75 students?

93
Typical Measures Associated with Quantitative and
Qualitative Methods

• Quantitative
• course exam, project, paper scores
• survey scores (Likert)
• frequencies of multiple choice test item
  responses
• scores on standardized scales and tests
• counts (participation, web requests, office
  visits)
• measures of time use
• institutional research data (GPAs, grades,
  admissions scores, frequency distributions of
  demographics)

• Qualitative
• performances (possibly recorded on tape)
• interviews (possibly recorded on tape)
• focus groups
• student projects, term papers, essay items or
  exams
• reflective statements
• journals
• reports of others (counselors, etc.)

94
Some Helpful Web sites
MCCs CASTL Group COPPER Cluster
Web www.middlesex.mass.edu/carnegie/ Explorations
from a Community of Practice http//www.middlesex
.mass.edu/carnegie/MCCCG/CGPublications.htm
COPPER Blog http//middlesex.blogs.com
Tutorial on the Scholarship of Teaching
Learning http//webcenter1.aahe.org/sotl_tutorial/
home.html Knowledge Media Lab Gallery
http//gallery.carnegiefoundation.org/
95
(No Transcript)