Workshop on the Table of Specifications

1
Workshop on the Table of Specifications

• Presented by
• Dr. Chan Chang Tik

2
Contents

• The Table of Specifications and the Bloom
  Taxonomy
• Construction of Examination Questions in line
  with the Learning Outcomes

3
Group Activity 1

• Discussion
• How do you know whether the examination questions
  you set are valid?

4
Valid?

• Moderators comments
• I think is a significant problem. The students
  cant answer the question properly on the basis
  of the information given . I consider the
  question is either unfair or unanswerable.
• Why it is here? I accept auditing is part of
  the learning outcomes but there seems to be too
  much auditing.

5
Test Validity

• To be valid a test must measure what it is
  intended to measure.
• Three points to consider in test validity
• Validity is specific to some particular use
• Validity is a matter of degree
• Validity pertains to the results of the test

6
Types of Validity

• Content validity
• Criterion-related validity (concurrent and
  predictive)
• Construct validity
• Which one to choose depends on the aims of your
  testing.

7
Content Validity

• The test should cover a cross section of the
  course.
• Emphasize important topics but give less
  attention to less important topics.
• The relative importance of topics in a test is
  directly proportional to the instructional
  emphasis given to the topics.

8
Table of SpecificationsStudents Response Q
Why?

• Step 1
• Number of lectures/chapters/units
• Number of questions in the final exam
• One table of specifications for each question

9
Step 1 Cont.

• 17 lectures in a course and final exam consists
  of 5 questions
• Q1 3 lectures Equally distributed?
• Q2 4 lectures
• Q3 3 lectures Grouping
• Q4 4 lectures
• Q5 3 lectures Discussion

10
Step 2

• Learning objectives
• Instruction
• Assessment
• Example (Mat250, Q1)
• List down the number of lectures
• List down the instructional objectives

11
Step 3

• Blooms taxonomy of cognitive skills
• Knowledge remembering facts, terms,
  definitions, and concepts. What?, list, name,
  define, describe
• Comprehension understanding the meaning of
  material. Explain, interpret, summarize, give
  examples, predict, translate

12
Blooms taxonomy

• Application selecting a concept or skill and
  using it to solve a problem. Compute, solve,
  apply, modify, construct
• Analysis breaking material down into its parts
  and explaining the hierarchical relations. How
  does apply?, Why does work?, How does
  relate to ?

13
Blooms taxonomy (2)

• Synthesis producing something original after
  having broken the material down into its
  component parts.
• How do the data support ?
• How would you design an experiment that
  investigates
• What predictions can you make based upon the
  data?

14
Blooms taxonomy (3)

• Evaluation making a judgment based upon a
  pre-established set of criteria.
• What judgments can you make about ?
• Compare and contrast criteria for ?
• Refer to the table of specifications

15
Table of Specifications Selected-Response Q

• Work as a team, for example each lecturer is
  assigned 5 lectures to come up with 50 multiple
  choice questions.
• Store any extra questions in the question bank.

16
Why Table of Specifications

• External Examiners Comments
• Several topics covered in the lecture programme
  are not assessed by this exam. My analysis would
  indicate no assessment of The Netherlands,
  France, Poland, Singapore, South Korea, Hong Kong
  (10 of the 28 weeks of teaching, 35.7).
• The module descriptor, the lecture programme and
  the exam do not tie up very well.

17
Why Table of Specifications (2)

• INTI-UC lecturers comment

I just want to say a million thanks for teaching
us how to do the Table of Specifications in
setting exam questions. It is only now that I am
able to appreciate how useful that table is,
now that I have managed to create my first set
of final exam questions (deadline today lah!). I
use the table and I am happy with the questions I
set because I could justify them if I were
asked. I think this part of the skill training
(the Table of Specs part ) should be made
compulsory for all new lecturers.
18
Table of Specifications Students Response
Questions

• Consider Question 1 of MAT250. There are 4 parts
  in this question
• taken from the following lectures
• Learning Outcomes
• Lecture 1 To define real numbers.
• To describe exponents and to produce the laws
  of exponents.
• To explain and simplify radicals.
• Lecture 2 To define the various types of
  equations.
• To relate linear equation to its
  applications.
• Lecture 3 To solve quadratic and radical
  equations.
• To discover some practical problems of
  equations.
• To explain some coordinates geometry topics.
• Lecture 4 To define inequalities.
• To solve linear, quadratic and rational
  inequalities.
• To solve equations and inequalities involving
  absolute values.
• Lecture 9 To produce partial fractions.

19
Table of Specifications (2)
Bloom 3
20
Table of Specifications Selected Response
Questions
21
Group Activity 2

• Set three questions each at different level of
  the Blooms Taxonomy using a table of
  specifications. Indicate clearly your learning
  outcomes.
• Present your questions to the participants.
• Time 45 minutes

22
Types of Assessment

• Essay
• Mathematical and technical problem items

23
Essay

• Use clear and concise questions
• Measure effectively complex learning outcomes
• Establish scope of expected response
• Handling of technical terms
• Content versus mechanics of expression
• Use descriptive words of precise meaning as
  directions
• No optional items

24
Clear and Concise Questions

• Language usage and word choice are particular
  important during question construction.
• The language dimension is critical because it
  controls the comprehension level of the item for
  the students and also it specifies the question
  parameters.

Example
25
Clear and Concise Questions (2)

• For example, Describe the origins of World War
  I
• A better question is What were the principal
  diplomatic events in Europe between 1890 and 1913
  that contributed directly to the outbreak of
  World War I
• You have to narrowly specify, explicate, define,
  or otherwise clarify what it is that you want
  from the students.

26
Clear and Concise Questions (3)

• The question set must have an answer that
  expert could agree upon, thereby rendering it
  objective. For example, Describe the world and
  give two examples.

Example
27
Complex Learning Outcomes

• Complex learning outcomes can be measured
  effectively with essay questions. These are the
  abilities to
• Explain cause-effect relationships
• Describe applications of principles
• Present relevant arguments
• Formulate tenable hypotheses
• Formulate valid conclusions

28
Complex Learning Outcomes (2)

• These are abilities to
• State necessary assumptions
• Explain methods and procedures
• Produce, organise, and express ideas
• Integrate learning in different areas
• Evaluate the worth of ideas

29
Extended versus Restricted Response

• From the instructors standpoint, an extensive
  response to a few broadly based questions allows
  an in-depth sampling of a students knowledge,
  thinking processes, and problem-solving behaviour
  relative to a particular topic.
• The open-ended nature of the question posed by
  the instructor such as Discuss the bond and
  share markets in the Malaysian context is
  challenging to a student.

30
Extended versus Restricted Response (2)

• Extensive response question is poorly structured
  and its free-response essay item tends to yield a
  variety of responses from the students, in both
  content and organisation, and thus inhibit
  reliable grading.

Example
31
Extended versus Restricted Response (3)

• A restricted-response question is preferable. For
  example, Discuss the advantages and
  disadvantages of essay and short-answer question
  with respect to (a) reliability, (b) objectivity,
  (c) content validity, and (d) usability.
• This question presents a better defined task
  which lends itself to reliable scoring, yet
  allows students sufficient latitude to organise
  and express their thoughts creatively.

32
Content versus Mechanics of Expression

• Factors such as expression, grammar, spelling,
  and the like can be evaluated in conjunction with
  contents of the essay.
• Evaluation of such skills is legitimate provided
  they are included in the learning outcomes.
• You may point out such mistakes in the students
  coursework.

33
Directive Words

• Compare
• Defend
• Trace
• Contrast
• Explain
• Differences

34
Non-directive Words

• Review
• Tell all that you know
• Report your knowledge

35
Classic Case

• Question (10 marks)
• Tell me all you know
• about Malaysia.
• Answer
• It is a beautiful country.

36
Example for Objectivity of Question

• External Examiners comment
• The students cant answer the question properly
  on the basis of the information given as they
  dont know how to spread the finance charges and
  looking at the three main alternative methods is
  not part of the question. I consider the question
  is either unfair or unanswerable.

37
Example of Extensive Response

• The relevance and the effects of accounting
  practices, have been the subject of much debate.
  Discuss the impact on the accounting practices.
• External Examiners comments
• Give source.
• I dont think this is a clear question and I
  would be amazed if any student produced a
  response anything like the outline answer.

38
Example of Language Usage

• A business or organisation can be positioned as
  effective as products and services.
• Internal examiners comment
• The sentence does not make sense. I suggest
  Successful marketing of products and services
  often depends on effective positioning in the
  market place.

39
Group Activity 3

• Look back at the essay or short-answer questions
  you set in Group Activity 2. Do you think you
  need to make any adjustments?
• Present your questions to the participants.
• Time 20 minutes.

40
Mathematical and Technical Problem Items

• Provide enough information and directions so that
  students clearly understand the problem.
• Indicate the degree of precision expected in the
  response.
• Clearly specify the units for reporting the
  response.
• When using regular everyday words to present a
  problem, make sure the connection between the
  structured situation and the real world is
  genuine.

41
Enough Information

• During the vacation Sue and Tai agreed to paint
  their house. One side of the house is
  eight-meters long. They mark a point on the side
  that is four meters from each other. Each one
  starts at one end and paints toward the center
  mark, four meters from each other. Can we be sure
  that each painted the same area of that side of
  the house if each paints to the center mark?

42
Degree of Precision

• In the following set of exercises for calculating
  z-scores from the raw scores, use the mean and
  standard deviation values to the nearest
  hundredth of a raw score unit.
• The proportions that you report for the ratios
  should be given in decimal fractions to the
  nearest hundredth.

43
Units

• Total elapsed time should be reported in minutes
  and seconds.
• When reporting the perimeters, give your
  responses in meters.

44
Everyday Words

• What words can you make from the letters in TEA?
• TEA, ATE, EAT, ETA, AET, TAE
• Create a permutation for the three letters in TEA.

45
Group Activity 4

• Look at the mathematical and technical questions
  set in Group Activity 2. Are the questions
  ambiguous? Do you want to re-set them?
• Present your questions to the participants.
• Time 20 minutes

46
Thank you