Tackling the Challenges to Professional Education Project: Reforming Professional Education for a Knowledge-based Society

1
Tackling the Challenges to Professional Education
Project Reforming Professional Education for
a Knowledge-based Society

• Using Criterion-Referenced Assessment to promote
  student learning
• Dr. Charles C. Chan, APSS, HKPU

2
Assessing student learning outcomes

• Who teachers, instructors, peers and even
  students themselves can administer the assessment
  of learning outcomes.
• For whom students (useful feedback for further
  improvement formative evaluation), instructors
  (formative and summative evaluation), and school
  or university administration (summative and
  quantitative evaluation).
• What subject and content specific, and/or
  generic transferable skills performance at
  practicum.
• How subjective grading, GPA, CRA taxonomy of
  educational objectives (cognitive) and behavioral
  competency.

3
Criterion-referenced assessment

• Judges performance against a set of
  pre-determined criteria
• Is independent of any other students results
• Increase students intrinsic motivation and their
  sense of being responsible for their learning
  outcomes
• Encourages the cooperation between students
• Improves the effectiveness of teachers feedback
  to students.

4
Comparison between NRA and CRA
Dimensions NRA CRA
Assessment method Compare with peers Mark with pre-set criteria
Feedback Summative Formative
Distribution Normal Skewed
Benefits Easy to administer Save time. Clear feedback Provide guidelines for further improvements Increase students intrinsic motivation.
Limitations Discourage cooperation among students Limits the effectiveness of feedback Increase students anxiety. Time-consuming Not widely used.
5
CRA The SOLO taxonomy (Biggs Collis, 1982)
SOLO level Characteristics of responses
Prestructural Irrelevant information is included or the task is not tackled incorrectly
Unistructural One relevant information is included
Multistructural More than one independent information are presented
Relational Various relevant elements are integrated that an organized picture is presented
Extended Abstract The original coherent picture is generalized to a new and more abstract domain, reflecting a higher mode of processing
6
CRA Behavioral Competence Taxonomy
Professionalism Knowledge Practice Competence
Acquisition Show awareness towards major societal problems Recognize social welfare policies Observe and listen skillfully
Assimilation Self-management personal growth Understand social, cultural factors to specific client system Specify goals of interventions
Adaptation Manage workload in an orderly manner Constructive use of available community resources Monitor and co-ordinate different work of the action plans
Performance Accept advice and criticism Apply theories learnt with flexibility Handle emergencies effectively
Aspiration Incorporate new approach to practice Initiate to expand ones knowledge base Recommend future improvements
7
Project I

• Subject Social Psychology (level 3) 01-02
• Participants 35 undergraduate students (yr1-3)
• CRA used the SOLO taxonomy
• Details Students are required to write three
  short papers with subject-specific scenarios and
  questions provided (one baseline, one mid-term
  and one final). The SOLO taxonomy is used to
  analyze students assignments and students are
  informed of their SOLO levels as feedback of
  their learning outcomes after each assessment.

8
Project I Assignment sample

• Short Paper I (500 words).
• You and one of your secondary school friends were
  successful to get into a degree program in the HK
  PolyU and the CityU / HKU respectively after one
  failure attempt. Right after both of you received
  the rejection letters, you met each other and
  found that the two of you had very different
  emotions.
• Q.1 Right after that failure attempt, recall what
  you and your friend came up with separately to
  account for that rejection.
• Q.2 Compare and contrast those reasons that were
  from you and from your friend and whether these
  reasons explained the different emotional states
  you were in.
• Q.3 By using theories-in-use, demonstrate to your
  friend that you understand why s/he thought the
  way s/he did.
• Q.4 Critically analyze why your friend agreed /
  disagreed with your perception of why s/he felt
  the way s/he did.

9
Project I Record sheet of SOLO level
Name of student Paper 1 Paper 2 Paper 3



10
Our Project I - Results

• Student learning outcome
• 2.54 (SOLO 1) 3.00 (SOLO 2) 3.25 (SOLO 3)
• Correlates with instructors judgment/subjective
  ratings.
• Undergraduates generally produce
  multistructural level papers.
• Students gradually improve during the semester.

11
Our Project I

• Instructors comments

12
Our Project II

• Subject Social Work Fieldwork I, II
• Participants 103 Social work undergraduates
  (including both bachelor and diploma students)
  40 fieldwork supervisors
• CRA used Behavioral Competence Taxonomy
  (self-developed, based on Hauenstein model and
  APSS fieldwork manual)
• Details fieldwork supervisors rated students
  performance in practicum with BCT (baseline,
  mid-term and final).

13
Our Project II Record Sheet

• Please identify the level of behavioral
  competence

• Student _________________________
• Supervisor _______________________
• Period under supervision ________________________
  ___________
• Name of the agency ______________________________
  _________
• Nature of the setting ___________________________
  ____________
• Nature of the major pre-selected piece of
  professional practice __________________________
  ___________________________________________
• Please check the following box to identify the
  nature of the major practice that requires the
  student to perform during the time period under
  review (Choose either one)
• ? 1. Require execution of relatively tangible,
  routine and well-defined practice procedures
• ? 2. Require re-negotiation, re-definition and
  problem-solving skills in the execution of
  practice procedures
• Remark __________________________
• may include planning and execution of a service
  program for adolescents, supportive counseling
  for pregnant teens etc.

Prof Knowledge PC
Level
Illustration

• 1 acquisition 2 assimilation
• 3 adaptation 4 performance
• 5 aspiration

14
Our Project II - Results

• Students improve over the semester

Baseline Mid-term Final
Professionalism 1.79 2.79 3.42
Knowledge 1.39 2.55 3.16
Practice Competence 1.43 2.84 3.50
15
Our Project II - Results

• Baseline
• Student is trying to acquire the knowledge of
  service and the organization.
• Mid-term
• Shows potentiality in integrating her experience
  and practice. Asks more sensible questions and
  more responsive in supervision.
• Final
• Student tried to apply social work values and
  principles into practice e.g. confidentiality of
  cases. Student was getting more confident in
  working with different client group.

16
Our Project II - Supervisors comments

• It takes quite some time to be familiar with the
  instruments.
• BCT offers more objective assessment than the
  traditional impression grading system.
• Its good to see students improve gradually over
  the semester.
• Sometimes it is difficult to decide which level
  students attained.
• Some placement setting (the nature) may limit
  the level of attainment a student can achieve.

17
Summary and Conclusion

• CRA benefits both students and teachers by
  providing valuable and objective feedback for
  continuous improvement in learning outcomes.

18
References

• Biggs, J. B. Collis, K. F. (1982). Evaluating
  the quality of learning the SOLO taxonomy
  (structure of the observed learning outcome). New
  York Academic.
• Chalmers, D. Fuller, R. (1996). Teaching for
  Learning at University Theory and Practice.
  London, UK Kogan Page Ltd.
• Chan, C.C., Tsui, M.S., Chan, M., Hong, J.
  (2002). Applying the SOLO Taxonomy on studetns
  learning outcomesL an empirical study. Assessment
  Evaluation in Higher Education, 27,6,511-528
• Chan, C. C. Cheng, G. H.-L. (2002). Using the
  SOLO Taxonomy to assess students written work.
  Submitted to Teaching of Psychology.
• Chan, C. C., Cheng, G. H.-L. and Chan, M. Y.-C.
  (2002, June). Assessing APSS Students Learning
  Outcomes and Learning Process. Teachers Manual
  version 2.1. (Available from the Network for
  Health and Welfare Studies, Department of Applied
  Social Sciences, the Hong Kong Polytechnic
  University)