Test Writing: Moving Away from Publisher Material

1
Test Writing Moving Away from Publisher Material

• Bruce Butterfras, MS Ed., L.P.
• Assistant Professor Dept. of Emergency Health
  Sciences
• University of Texas Health Science Center at San
  Antonio

2
Objectives

• Explain the importance of an EMS curriculum
  unique to your program
• Identify the steps involved in creating good
  exams
• Create and evaluate exams for your courses

3
Why Customize My Course?

• EMT/Paramedic National Standard Curriculum
• From 1971 to 2000
• Contained specific knowledge and skills
  objectives
• EMS Education Agenda for the Future (2000)
• National EMS Education Standards
• The EMS NSC, with their detailed declarative
  material, limit instructor flexibility and the
  ability to adapt to local needs and resources

4
Publisher Resources

• Most EMS textbooks come with instructor resources
• These often include chapter objectives, lesson
  plans, PowerPoint presentations and test banks to
  be used to construct exams
• When selecting a textbook these additional
  resources should be a strong consideration
• Disclaimer I have helped create Publisher
  material including PowerPoint presentations and
  test questions

5
Publisher Objectives

• Publishers create objectives based on the
  National EMS Education Standards
• The National EMS Education Standards are
  intentionally broad, so specific objectives vary
  among the publishers
• Publishers objectives are by necessity generic
  in order to be applicable to all areas of the
  country
• Often specific to their textbook as well

6
Publisher Lesson Plans

• Publisher provided Lesson Plans are also
  relatively generic in order to work for programs
  around the country
• These lesson plans usually focus on presentation
  of material (that is found in the textbook) to
  students in a standard (ie. lecture oriented)
  educational model

7
Publisher Presentations

• Publisher provided PowerPoint presentations
  highlight the material presented in the textbook
• This can allow students to follow along in the
  textbook as the lecture progresses
• Publisher provided PowerPoint presentations can
  also be
• Reorganized to fit your way of teaching the
  subject
• Added to with additional materials from your own
  research on the topic

8
Modifying Presentations

• Publisher provided PowerPoint presentations can
  also be
• Deleted from if you feel there is material
  included that doesnt need to be discussed
• Spiced up to make them more appealing to students

9
Publisher Test Banks

• Publisher provided test banks serve primarily to
  determine if your students remember what they
  read in the textbook
• The questions almost always refer to a particular
  page in the textbook
• You need to test your students on what you taught
  them and not just what is in the book

10
So What Do I Do?

• To make your class more effective, you need to
  modify publisher materials or create your own
• Go through the objectives and modify them to meet
  your needs. Add, delete, or edit objectives to
  fit your particular course
• Create your own presentations or edit those
  provided to make them fit your course
• Use other teaching techniques besides lecture!

11
Can I write a good exam myself?

• Finally create your own exams with questions you
  create to test what your students have learned
• Start by asking what type of exam will this be
• A short quiz to see if the students read their
  book
• A longer exam covering several chapters worth of
  material
• A final exam testing mastery of the entire course

12
Where do I start?

• Start by making a Blueprint for the exam
• Identify what information will be tested
• Select an exam length appropriate for the
  information and the purpose of the exam
• Determine how many questions will be included
  from each section

13
Am I testing what I should?

• Exams should test if the students can meet the
  course objectives
• So each test question should be tied to an
  individual objective
• Think about the objective as you write each
  question and make sure the question you write
  actually tests the objective

14
What question format is best?

• There are many types of test questions
• You need to choose the one (or ones) that best
  fit your situation (and/or your students)
• Essay Questions
• What are the advantages and disadvantages of
  Essay questions?

15
What about other types of questions?

• Short Answer / Fill in the Blanks
• Name one advantage of a short answer question.
• Fill in the blank exams are good for testing
  ______ of presented materials.
• True / False questions
• Is this statement true or false
• True / false questions are a favorite of students
  because they are always clearly written and give
  students a 50/50 chance of getting the answer
  right.

16
Can I use Matching questions?

• Matching Questions

1. Matching questions make this easy 2. Matching
questions are poor at 3. Matching questions are
good for 4. Matching questions are difficult in
this respect
a. Creating b. Evaluating understanding c.
Grading d. Testing recall
17
What about Multiple Choice?

• Multiple Choice Questions
• How many of you primarily or exclusively use
  multiple choice exams? Why?
• Multiple choice questions are
• A. The best choice for testing recall of
  information
• B. Easy to write as a test of advanced
  understanding
• C. Capable of evaluating higher levels of
  thinking
• D. A poor choice for evaluating general knowledge

18
Writing Multiple Choice Questions

• Since this is probably the most popular type of
  question used for EMS exams, lets look at how to
  construct a good MC question
• Multiple choice questions have three parts
• A Stem
• A Correct answer
• Several Distractors (incorrect answers)

19
Multiple Choice Questions

• Start with a Stem tied to one of the objectives
  being tested
• Try to create a stem that requires more than just
  recall (memorization of facts)
• Scenarios work well for these types of questions
• Include enough information to choose the correct
  answer and possibly additional irrelevant
  information

20
Multiple Choice Questions

• Write the Correct Answer
• Tie it to the objective and make sure that there
  is only one correct answer
• Avoid All of the Above and None of the Above
• You can use Both A and B type answers if you
  have enough choices (usually more than 4)
• Consider writing a feedback statement to
  explain why this answer is correct

21
Multiple Choice Questions

• Now write the Distractors
• Make each distractor plausible
• Try to use answers that might be given by
  students who do not have a good grasp of the
  material
• Make sure there is something in the distractor
  that makes it incorrect
• Consider writing a feedback statement telling
  why this answer is not correct

22
Multiple Choice Questions

• Distractors (continued)
• Make sure both distractors and correct answer
  sound correct (read each question out loud with
  each answer option)
• Make all answer choices almost the same length
• Shortest or longest answer choices often stand
  out
• If necessary reword the stem to make the answers
  fit
• Avoid NOT, EXCEPT, etc. in stems if possible

23
Multiple Choice Questions

• Now consider alphabetizing the answer choices
• Prevents the tendency to write 2 distractors, the
  right answer and then another distractor so the
  correct answer is most likely C
• Include enough questions to test the objectives
  covered in the material
• Somewhere within at least one exam there should
  be a question for each course objective

24
Building the Exam

• Use a variety of easy and difficult questions
• Some recall, some higher level thinking
• This will allow you to identify students at
  different levels of understanding
• It will also allow you to anticipate the grades
• Aim for a mean grade of 85 and half should be
  above and half below
• If they score below a 70 they need remediation

25
Testing the Exam

• When you finish your exam, test it out
• Give the exam to your fellow program instructors
• See what grades they make listen to their
  feedback
• Dont expect their grades to be perfect they
  didnt sit through your class
• Consider also giving the exam to a few recent
  graduates and listen to their feedback

26
Giving the Exam

• Finally you should be ready to give the exam
• Consider giving computerized exams if possible
• Allow about one minute per question
• Listen to the students feedback afterwards
• If there is another way to interpret your
  question, they will find it and point it out to
  you!
• Make adjustments to the questions based on
  student feedback
• Keep statistics on each question

27
Evaluating the Exam

• Each time before you teach the class review your
  exams and make adjustments
• Keep your own test bank of questions in case
  you need to create a make-up exam
• It will also allow you to use different exams
  each time you teach the class
• Always be open to suggestions for improvement

28
Conclusion

• Publisher provided materials are a great starting
  point, but dont rely on them exclusively for
  teaching your class
• Multiple choice questions can make great exams so
  practice making them whenever you can
• Constantly reevaluate your courses and strive to
  make them better. The life your student may
  eventually save may be yours!

29
Selected References

• National EMS Education Agenda for the future
  http//www.nhtsa.gov/people/injury/ems/edagenda/fi
  nal/agenda6-00.htm
• Cason, D. McKenna K. Foundations of Education
  An EMS Approach 2nd edition Clifton Park, New
  York Delmar Cengage Learning (2013) Print
• My Contact Information
• Butterfras_at_uthscsa.edu
• Office phone - 210-567-8719