Instructional Design Evaluation, Assessment

1
Instructional DesignEvaluation, Assessment
Design A Discussion (EDER 673 L.91 )From
Calgary With Asst. Professor Eugene G.
Kowchmarch 27(An Synchronous Meeting using
WebCT discussion Thread and WWW Course Home
Page Material)
2
Agenda

• Housekeeping
• Timetable
• Course Objectives
• Blueprint Guidelines (online)
• ID Model Guidelines (online)
• Assessment and Evaluation - a Quick Overview
• Discussion

3
Timetable

• Keep an eye on the home page -)

4
Update EDER 673
History of ID
ID Terminology
Instructional Design Philosophies
SMCR/Feedback Communication Model
Learners and Learning Theories
Context based designs
ID Models A peek
Ethics/ Ldrship
Needs Analysis
Media Selection
Task Analysis
Ordering Content (elaboration)
Evaluation
Motivation
5
Course Objectives Harvest Time

• 1. Explore theories of learning and
  instruction, exploring implications and possible
  applications for their own practice,
• 2. Explore, develop and articulate their own
  instructional design models, to see how theory
  and practice can be articulated,
• 3. Design and present efficient, effective
  and appealing instructional interventions
  informed and referenced to theory,
• 4. Participate and present in a collaborative
  learning community to exchange and consider
  developing design ideas as instructional
  designers,
• 5. Examine the potential and limitations of
  media and technology use
• 6. Analyze some models critically and
  instructional interventions associated with the
  selection and sequencing of content across the
  instructional spectrum and
• 7. Explore and analyze the matching of
  instructional strategies to characteristics of
  learners and content.

6
Blueprint Guidelines

• I. Approach ism
• II. Introduction State the purpose of this
  instruction.
• III. Audience State the intended audience for
  your report
• IV. The Performance Problem Explain the gap you
  will fill by your designed instruction.
• V. Learners Provide a description of the
  learners for whom this instruction is intended.
• VI. Objectives State the learning outcomes that
  you desire from this 30 minute instructional
  module.
• VII. Scope and Sequence Describe the decision
  making process you went through
• VIII. Indicate optimal and minimal requirements
  for media and technology
• IX. Indicate the instructional flow / process
• X. Conclusion

7
Blueprint Guidelines

• Grading Method for this project
• Format did the student follow the requested
  blueprint format? 10
• Content
• ID Model Application did the student include
  and explain how
• the instructional blueprint uses the student's
  personal ID model? 30
• Is the performance problem clearly
  identified? 10
• Clarity Coherence is the report clear and
  legible/logical? 20
• Scholarly / Theoretical basis Are key models and
  theories cited? 10
• Conclusion Does the conclusion explain the
  blueprint to someone who
• might contract the designer for exactly this
  instruction? 20
• total 100

8
Blueprint Presentation Guidelines 6 elements
9
Blueprint Presentation Guidelines 6 elements
10
Blueprint Presentation Guidelines 6 elements
11
Blueprint Presentation Guidelines 6 elements
12
Blueprint Presentation Guidelines 6 elements
13
Blueprint Presentation Guidelines 6 elements
14
Blueprint Presentation Guidelines 6 elements
See the bottom of the home page for guidelines
15
ID Model Guidelines

• Informed Do theories underpin the model?
• Referenced Are theorists referenced/understood?
  (APA not needed)
• Relationships Between Elements are
  indicated/explained
• Granularity Visual Model addresses most elements
  of models covered in EDER 673
• Missing Elements A rationale is provided for
  element exclusion in your model
• Limitations of the ID Model Are defined
• Generalizability Evidence or explanation of use
  of this model in
• 2 different content areas
• Scalability Can I understand the lesson and the
  program design via your model?
• Innovation Uniqueness - do you explain a new
  approach?

16
ID Model Guidelines
17
ID Model Guidelines
18
ID Model Guidelines
19
ID Model Guidelines
20
ID Model Guidelines
21
ID Model Guidelines
22
ID Model Guidelines
23
ID Model Guidelines
24
ID Model Guidelines
25
ID model Presentation (April 17)Guidelines.
26
ID model Presentation (April 17)Guidelines.
27
ID model Presentation (April 17)Guidelines.
Load file B here
28
Instructional DesignEvaluation, Assessment
Design A Discussion (EDER 673 L.91 )From
Calgary With Asst. Professor Eugene G.
Kowchmarch 27(An Synchronous Meeting using
WebCT discussion Thread and WWW Course Home
Page Material)
29
Assessment Instruments Assessing learners

• Criterion Referenced
• Pretest Placement entry level skills
• Readiness Questions
• Placement Questions
• Posttest helps us identify what did not work for
  this learner
• ---------------------
• To make these tests, we need a lot of info
• List of task based behavior objectives or
• Type of learning (verbal, etc, ) and
• Type of conditions for learning (Gagne) and
  events for learning
• Bad criteria test subjects
• Writing a paragraphaffective evaluations..

30
Pretest Method of Inquiry for Placement
From Driscoll, M. (2000). Psychology for
Instruction, 2nd Ed.
Placement Learner Readiness Assessment
(to determine entry performance)
Readiness ConcernDo the students have
the Prerequisite skills?
Placement Concern Have the students
already Achieved the intended Outcomes?
yes
no
yes
no
Provide readiness experiences
Proceed with The instruction
Advance Students To a higher level
Discuss Math vs. Social Studies (criterion?)
31
Formative and Summative Assessments( of your
Design)

• Formative methodology (can be done in order)
• Education Context 1 to 1 evaluation (designer
  learner)
• Questionnaires, interviews - check learning time,
  outcomes
• Focus group
• Nets out the instructor
• Check the effectiveness of instruction
  (viability, attitudes of instr.)
• Field Trial
• Observe a full instruction event with the 2x
  modified course materials and design test for
  achievement.

32
Formative and Summative Assessments( of your
Design)

• Summative Assessment methodology
• Expert Judgment (assess the instructor)
• Congruence Can the instructor meet the needs of
  the organization?
• Content Are the materials complete, updated?
• Instructor followed the design?
• Utility of instructor?
• Design Are the principles of learning,
  instruction and motivation in the materials?
  Instruction?
• Field Trial
• Outcomes Analysis impact to learner on job, in
  school
• Management Instructor, supervisor performance
  test

33
Summative Learner Assmt. Method
Summative Assessment (to monitor learning
progress)
Have the students achieved The intended
outcomes Of instruction?
no
yes
Provide additional learning experiences
Assign grades to certify mastery
Evaluate the effectiveness of the instruction
Discuss
34
Assessment Planning (Magliaro from notes)

• Assessment Learner performance
• Evaluation judging learner decisions
• Program Evaluation Appraising program success
• Assessment Possibilities
• Portfolios
• Physical
• Electronic
• Communicate task and knowledge performance
• Time consuming for all!
• Projects
• Demonstrate action / processes / competencies
  (training)
• Address problem solving (education)
• Require relevant subject matter!

35
Assessment Planning (Magliaro from notes)

• 5 Step assessment process
• Identify goals and objectives from Needs Assmt
• Match goals objectives with assmt tools
• Choose instructional method
• Design tasks and scoring methods (match)
• Think through the implications.
• Guidelines
• Start small
• Develop rubrics
• Communicate purpose of assessment
• Involve participants
• Keep going - recursive design
• Look for blips - unfairness, odd results

36
PostTest Method of Inquiry for Placement
The instructional role of Formative Assessment
During Instruction
(to monitor learning progress)
Are the students achieving the intended Learning
outcomes?
no
yes
Provide Group or Individual remediation
Provide feedback to reinforce learning
Proceed with The planned instruction
DIAGNOSTIC ASSESSMENT (to study persistent
difficulties
Discuss
37
Recall The one and the many
38
Norm
The Distribution Curve Defining Normal
Relative ranking, describes what cannot be
done And what can be done..
39
Great questions
Designers Ask..

• "Can and should instructional designers place
  importance
• on process instead of product? If yes, is this
• desirable and reversible? Discuss"

40
Great questions
Designers Ask..

• Should and can evaluation be both internal and
• external?
• Should and can the evaluator personalize his
• or her findings and put those results into a
  cultural
• perspective? Discuss."

41
Designers Ask..

• 1) There is the sense that "outside the company"
• designers are not as biased as "backyard"
  designers. I
• wonder whether this is in fact true, or whether
  their
• biases are simply different. The "outside"
  designers
• could have biases that are harmful or just as
  blinding
• as "in house" people's.

42
Evaluation/Assessment
Designers Ask..

• There seems to be a belief that objectivity is
• somehow better than subjectivity, and I wonder
  about
• that too. Sometimes, a subjective approach may
  be
• called for. For example, in evaluating a Native
• education program, would it be effective to use a
  white
• person? Maybe.
• Then again, maybe not. Maybe the white
• person would have to spend so much time and
  effort
• learning first about the culture that her work on
  actual
• evaluation would become miniscule in comparison.