Developing and Assessing the Research Skills of Students in Engineering Courses

1
Developing and Assessing the Research Skills of
Students in Engineering Courses

•  
• Dr Said Al-Sarawi
• Research Skill Development and Assessment
• ALTC Project Member
• School of Electrical and Electronic Engineering
• University of Adelaide
• A seminar for Curtin Engineering Faculty

2
Outline

• Motivation
• Students and staff issues
• RSD Framework in Australia
• Benefits of using RSDF
• What is RSD Framework?
• Case Studies From 1st Year to Master Level
• Other Dimensions of RSD Framework
• RSD at the Program Level !
• RSD in conclusion
• Discussion and future work

3
Motivation (1/2)

• PhD completion rate were doubled for students who
  had participated in undergraduate research (Bauer
  Bennett, 2003)
• Performance based research funding (old RQF,
  ERA)
• Performance based learning and teaching support
  Learning and Teaching Performance Fund (DEST,
  2006)
• Undergraduate research has been associated with
  higher level of student satisfaction and generic
  skills development (Kardash, 2000)
• Factors affecting skills implementation are
  (Lucas et al, 2000)
• Scepticism of the message, the messenger and its
  vocabulary
• The skills demanded lack clarity, consistency and
  recognisable theoretical base
• The skills are dependent on discipline area

4
Motivation Student and Staff issues (2/2)
Staff Student
How can I develop and assess students research skills? What is meant by Research and what skills are needed?
Is there a systematic, explicit approach that can be used? How I can acquire and develop these skills?
Is this issue only limited to the engineering discipline? How are they relevant to my career aspirations?
How can I remove the subjectiveness from the assessment? How will my skills be assessed?
5
RSDF in Australia

• The University of Adelaide (John Willison)
• Human Biology (Eleanor Peirce  Mario Ricci)
• Electrical Engineering Masters by Coursework
  (Said Al-Sarawi and Brian Ng)
• Clinical Nursing (Frank Donnelly)
• Petroleum Engineering (Steve Begg)
• Introductory Academic Program (Richard Warner)
• English (Joy McEntee)
• Dentistry (Vicki Skinner and Leonard Crocombe)
• Oral Health (Sophie Karanicolas and Cathy
  Snelling)
• Software Engineering (Li Jiang)
• Veterinary Science (Susan Hazel)

6
RSDF in Australia (Conts.)

• Macquarie University (Psychology Judi Homewood)
• Monash University (Business Ethics Jan Schapper
  Sue Mayson Business Glen Croy Tourism)
• University of Melbourne (Business Law Eu-Jin
  Teo)
• University of South Australia (Introduction to
  Tertiary Learning, 2008 Rowena Harper)

7
(No Transcript)
8
The facets of student research

• In researching, students
• embark on an inquiry and so determine a need for
  knowledge/understanding
• find/generate needed information using
  appropriate methodology
• critically evaluate information/data and the
  process to find/generate
• organise information collected/generated
• synthesise and analyse and apply new knowledge
• communicate knowledge and the processes used to
  generate it, with an awareness of ethical, social
  and cultural issues.
• (Willison ORegan, 2007)

9
Learning to Frame Research Questions

• Level 1
• Facet A) Embark on Inquiry
• Respond to questions / tasks arising explicitly
  from a closed inquiry.
• Facet E) Synthesis, analysis, application Ask
  questions of clarification / curiosity.

10
Learning to Frame Research Questions

• Level 2
• Facet A) Embark on Inquiry
• Respond to questions / tasks required by and
  implicit in a closed inquiry.
• Facet E) Synthesis, analysis, application Ask
  relevant, researchable questions.

11
Learning to Frame Research Questions

• Level 3
• Facet A) Embark on Inquiry
• Respond to questions / tasks generated from a
  closed inquiry.
• Facet E) Synthesis, analysis, application
• Ask rigorous, researchable questions based on
  new understandings.

12
Why use RSD approaches? (1/2)

• Benefits of using the RSD for students (according
  to Eleanor Peirce and Mario Ricci, Medical
  Sciences, Uni of Adelaide)
• Their research skills in our course have
  improved.
• They understand much more clearly what is
  expected of them.
• They know exactly where they need to develop,
  thanks to feedback.

13
Benefits for Lecturers (2/2)
(according to Eleanor Peirce and Mario Ricci,
Medical Sciences, Uni of Adelaide)

• We can give feedback on assessment tasks more
  accurately and efficiently we can give the same
  quality feedback with less writing, and faster.
• We can get a much better idea of where our
  students are from a quick analysis of the RSD
  results than from a detailed analysis of standard
  marks.
• We can easily match assessment tasks with course
  objectives, and course objectives with the
  Universitys Graduate Attributes.

14
Case Study 1 Human Biology - 1st Year

• Read the two short then complete tasks 1 and 2.
• Task 1
• Integrate the information presented in the two
  articles to write your own dot-point notes
• on the worksheet attached. To do this
• Identify 3-4 key ideas from the articles
• Use these key ideas to formulate headings and
  underline each
• Make bullet-point notes and list them under these
  headings.
• After each point, indicate its source, i.e.
  whether the idea came from article 1, article 2,
  or both
• Provide a title that embodies the content of
  your notes.

O Week RSD O Week RSD O Week RSD
I II
A


F
Task2 Which of the two articles do you consider
to be the better source? On what
characteristics/features of the article have you
based your choice?
15
Case Study 1 Human Biology - 1st Year
16
Case Study 1 Human Biology - 1st Year
Literature Research Skill Stream
O Week RSD O Week RSD O Week RSD
I II
A


F
Small Group Inquiry Small Group Inquiry Small Group Inquiry Small Group Inquiry Small Group Inquiry
I II III IV
A


F
Lit Week RSD 1 Lit Week RSD 1 Lit Week RSD 1 Lit Week RSD 1
I II III
A


F
Lit Week RSD 1 Lit Week RSD 1 Lit Week RSD 1 Lit Week RSD 1
I II III
A


F
Laboratory Research Skill Stream
Individual Open Inquiry Individual Open Inquiry Individual Open Inquiry Individual Open Inquiry Individual Open Inquiry
I II III IV
A


F
Lab RSD 1 Lab RSD 1 Lab RSD 1
I II
A


F
Lab RSD 2 Lab RSD 2 Lab RSD 2
I II
A


F
Field and literature research
Semester 1
Semester 2
17
Novelty in the engineering case

• The differentiation points
• Clearly state the need to research skill
  development
• The students are required to identify a gap in
  knowledge for each of the chosen topics (the
  unknown)
• Rigorous literature research

18
Case study 2 Photonic and Communication Master
Course

• Masters (coursework) course at EEE
• Course run by senior colleague in EEE
• 2 units of lectures/exams
• 1 unit of literature research project
• Students seek supervisors individually
• Topic chosen by student, but requires approval by
  supervisor
• Demographics
• Almost all are international students usually
  East Asian background
• Technically capable, but generally lack prior
  experience in conducting research
• Goal produce a high quality review paper on
  chosen topic
• Mostly negative experiences in 2005
• Cohort lacked basic literary research skills
• Unstructured approach towards project
• Low quality final review papers

19
Process Details

• Initial diagnostic task
• Supervisor supplies two technical papers
• Different levels (magazine, journal) for contrast
• Students summarise and compare the key points
  from both sources in one structured, bulleted
  list
• Detailed supervisor feedback in written form
• Knowledge accumulation phase
• More articles added to the reading list
• Continually add to an organic structured,
  bulleted list
• Strategy on further reading
• Student applies critical evaluation on
  suitability of sources with supervisor input
• Fortnightly workshops for group presentations
  discussions
• Supervisor supplies feedback optional external
  advice (CLPD)
• Writing phase
• Student writes review paper based on list
• Supervisor feedback on first draft approx a week
  before submission

20
Outcome and Evaluation

• Outcomes from 2006
• Total of 6 students plus1 external (industry)
  student as control
• Qualitatively, much improved papers compared to
  2005
• Coherent structures, logical arguments,
  conciseness, respect of referencing practice
• Student perspective
• struggled to cope with demands of research among
  the pressures of regular coursework
• English as second language remained a great
  barrier
• Framework matrix useful for quantitative
  assessment
• Unexpected benefit reduction in plagiarism
• Turn-it-in software reported major improvements
  from 2005
• Reports tend to be on conservative side.

21
Case 3 Final Year Project (in Progress)
We have been concerned about a degree of
disconnect between the desired leaning outcomes
and the assessment methods used in these
projects. HoS of School of EEE, The University
of Adelaide.
The set of project deliverables are The set of project deliverables are
Semester 1 Formative literature assessment (diagnostic value?) Proposal seminar design document/interim report Critical design review (peer review)
Semester 2 final report final seminar project log books/wiki/blog/email log etc
22
RSD Useful First-year to PhD?
23
Variety of RSD approaches

• Five distinct approaches have emerged in the use
  of the RSD so far
• Rubrics base to assess the profile of skills for
  each student, as demonstrated by Human Biology
  rubrics.  this is the most common approach, and
  is used by most disciplines.
• A lock-step approach, whereby students are kept
  'in formation' and progressively and corporately
  develop one level of skills at a time.  This is
  shown by the Nursing RSD rubrics.
• Grading with a specific grading, by incorporation
  into SOLO taxonomy to define grading within a
  specific level set, as used by Dentistry.
• Skill Evaluation, to evaluate the skills and
  levels required by existing assessments.
• Identification of research skills of higher
  degree students, Masters by Research and PhD
  students and/or their supervisors/advisors to
  locate their present skill set and plot future
  directions and development needs. 

(SOLO Structured Observed Leaning Outcomes)
24
Other Dimensions of RSD

• Degree of Autonomy
• Degree of Academic Rigour
• Degree of Conceptual Demand
• Degree of disciplinary knowledge required
• Status of knowledge being pursued

25
RSD Framework at Program Level !
No studies that consider student outcomes of the
explicit development and assessment of research
skills over a whole undergraduate or
masters-by-coursework program

• Undergraduate Level
• Bachelor Oral Health, Adelaide Uni, AU
• Bachelor of Media Studies , Adelaide Uni, AU
• Bachelor of Science, Adelaide Uni, AU
• Bachelor of Nursing, Adelaide Uni, AU
• Bachelor of Business, Monash Uni, AU
• Bachelor of Science (Psychology), Macquarie
  University, AU
• Postgraduate Level
• Master of Engineering (Advanced) (Electrical),
  Adelaide Uni, AU
• Bridging Program for International PhD students,
  Adelaide Uni, AU
• Masters and PhD (Nursing), PhD (Nursing)
• Trinity College Dublin, Ireland
• Master of Business, Monash University, AU

26
RSD in conclusion

• Provides the Big Picture and relates this to the
  assessment details for course coordinators,
  lecturers, tutors, and especially students
• Informs assessment-first curriculum redesign
• Same facets for multiple assessments, various
  levels
• Explicit Transparent assessment criteria
• Coherent Incremental skill development
• Revisited (potentially) Cyclic Conceptual
  Framework

27
Discussion and Future work

• How to integrate the framework into other
  Programs?
• How will the implementation of RSD framework
  affect academic workload?
• How to monitor and assess students progress?
• How can this be implemented for larger class
  sizes?
• How can the framework adopted for non-literary
  research skills?

28
References
Acknowledgement
This session was funded by an Australian Learning
and Teaching Council Grant
RSD Web Site
http//www.adelaide.edu.au/clpd/rsd

1. Bennett, N., Dune, E. Carré, C. (2000). Skills
   Development in Higher Education and Employment.
   (Society for Research into Higher Education
   Open University Press).
2. Stevens, C. Fallows, S.J. (2000). Integrating
   Key Skills in Higher Education employability,
   transferable skills and learning for life.
   Routledge, ISBN 0749432659.
3. Lucas, U., Cox, P., Croudace, C. and Milford, P.
   (2004). Who Writes This Stuff? Students
   Perceptions of Their Skills Development. Teaching
   in Higher Education, 9(1), 55-68.
4. Willison, J.W. ORegan, K. (2007). Commonly
   known, commonly not known, totally unknown A
   framework for students becoming researchers.
   Higher Education Research and Development, 26(3).