A u s t r a l i a n U n i v e r s i t i e s Q u a l i t y A g e n c y

1
Rob Carmichael
Learning about LT from AUQA audit findings
Evaluation Forum 2006 29 November
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
2
Session Outline

• Acknowledgement Background
• Learning from AUQA Commendations
• Learning from AUQA Recommendations
• Commendations v Improvement insights
• Conclusions What can educational development /
  LT development units do?
• A closing word on AUQA Cycle 2 Audits

2
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
3
1. Acknowledgement Background

• This study builds on work done by Alex Radloff
  of CQU Barbara de la Harpe of RMIT for their
  HERDSA 2006 paper Insights about learning and
  teaching from AUQA commendations, which the
  author gratefully acknowledges.
• Radloff de la Harpe subjected the
  Commendations for good practice from 24 AUQA
  audits to content analysis. AUQA performs
  fitness-for-purpose audits using the
  Approach-Deployment-Results-Improvement (ADRI)
  method.

3
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
4
O (Objective)
I (Improvement)
R (Results)
A (Approach)
D (Deployment)
4
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
5
Background (cont.)

• AUQA makes three types of audit findings
• Commendations recognising good practice
• Recommendations identifying areas for
  improvement, and (more recently)
• Affirmations, which are a subset of
  Recommendations acknowledging areas of
  improvement which have already been identified by
  the university in its self-review.
• This study categorised the Recommendations
  following the Radloff de la Harpe methodology.
• Because Affirmations were introduced
  part-way-through the period, for the sake of
  consistency they were not included in this study.
• As at today, all but one Australian university
  has been audited by AUQA.

5
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
6
2. Learning from AUQA Commendations

• 328 Commendations from 24 audit reports available
  at January 2006 were subject to categorisation
  using content analysis methods.
• Definition of LT any activities, processes or
  initiatives aimed at supporting student learning
  and enhancing teaching at both UG and PG levels.
• 125 (38) of all Commendations were categorised
  as falling within this definition.

6
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
7
Learning from AUQA Commendations (cont.)

• Within LT, six distinct categories were
  identified and the Commendations were allocated
  using a bottom up approach following Merriam
  (1990)
• Physical resources infrastructure
• Social climate activities that encourage
  support students
• Intellectual activities that support
  intellectual engagement
• Curriculum program structure, mode, design
• Teaching professional development roles, and
  initiatives
• Quality systems and processes aimed at quality
  assurance

7
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
8
Learning from AUQA Commendations (no.)

• Physical (6) majority of findings were for
  infrastructure. Others campus environment
  library resources.
• Social (30) majority for provision of special
  support programs for students responsiveness to
  student needs.
• Intellectual (18) majority for graduate
  attributes and academic skills programs.

• Curriculum (13) majority for program structures
  for flexible learning pathways. Some also for
  good program design.
• Teaching (31) majority for prof. development,
  especially mentoring induction for new staff,
  also incentives grants, leadership schemes.
• Quality (35) majority for QA review processes
  for courses, progs. Org. units, for student
  feedback systems.

8
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
9
Commendatory Insights gained

• In order of category magnitude
• Quality universities are acknowledging the
  importance of ensuring LT quality and have some
  well-developed practices and processes to
  monitor student perceptions and feedback. Need
  to now strengthen focus on standards and
  benchmarking.
• Teaching recognition of the importance of PD
  Activities for academic staff to support LT. A
  number of academic / LT development units
  commended for their role, but observation that
  such centres may often be marginalised, their
  role questioned and face restructure budget
  cuts.
• Social good practices in special support
  programs to improve student engagement. Need to
  now evaluate the impact of existing practices.

9
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
10
Commendatory Insights (cont.)

• In order of category magnitude (cont.)
• Intellectual work done around graduate
  attributes has paid off in identifying and
  embedding them into the curriculum. Further work
  now needed to ensure that once embedded they are
  taught and assessed. Need for initiatives aimed
  at supporting academic skill development.
• Curriculum some good practices in award
  structures and pathways for students with a focus
  on increasing flexible learning. Need for a
  learner-centred, constructivist approach to
  curriculum design across disciplines.
• Physical small number of Commendations
  symptomatic of financial pressures facing
  universities?

10
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
11
Commendatory Insights (cont.)

• Overall, Radloff de la Harpe concluded that
  Australian universities were able to evidence of
  exemplars of good practice related to supporting
  learning and teaching, suggesting that the sector
  is embracing the quality agenda and is responding
  to improve LT quality.
• However, these researchers also highlighted the
  need for universities to do more to support
  intellectual development and good curriculum
  design. They urged the adoption of a a
  student-centred approach to learning with a focus
  on intellectual development (e.g. peer academic
  mentoring), etc.

11
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
12
3. Learning from AUQA Recommendations

• 431 Recommendations from 24 audit reports
  available at January 2006 were subject to
  categorisation using the same definition of LT
  and the Radloff de la Harpe LT sub-categories.
• 220 (51) of all Recommendations were categorised
  as falling within the LT category.

12
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
13
Learning from AUQA Recommendations (no.)

• Physical (21) majority of Recommendations were
  to improve resources for LT. Others
  Infrastructure Library resources systems.
• Social (30) majority to improve campus climate
  responsiveness to students. Other Mentoring
  induction and student support programs.
• Intellectual (27) majority for academic enabling
  skills programs, followed by graduate attributes.

• Curriculum (33) about equal between improving
  program structures for flexible learning
  pathways, and improving program design (the two
  being closely related).
• Teaching (32) majority for clarifying and
  supporting the roles performed by Academic staff,
  followed by Prof Dev (again the two being closely
  related).
• Quality (76) majority to improve systems for
  student feedback on LT quality. NB. Very few
  recommendations mentioned standards and
  benchmarking.

13
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
14
Comparison of findings across Categories as a
(no.)

• Commendations
• Quality 26 (35)
• Teaching 23 (31)
• Social 22.5 (30)
• Intellectual 13.5 (18)
• Curriculum 10 (13)
• Physical 4.5 (6)

• Recommendations
• Quality - 34.5 (76)
• Curriculum 15 (33)
• Teaching 14.5 (32)
• Social 13.5 (30)
• Intellectual 12 (27)
• Physical 9.5 (21)

14
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
15
4. Commendations v Improvement Insight Gained

• Recommendations outnumber Commendations at a
  ratio of 54.
• There was a higher proportion of Commendations in
  Social, Teaching, and Intellectual (marginal).
• There was a higher proportion of Recommendations
  in Quality, Physical, and Curriculum.
• Q. Are Commendations Recommendations mutually
  exclusive categories?

• Comparison of the two sets of findings is broadly
  consistent for Teaching, Social, and Curriculum,
  is reasonably neutral for Intellectual, and there
  is some inconsistency for Physical (but smaller
  numbers involved).
• The biggest inconsistency is in Quality, which
  attracted the largest of Recommendations
  (34.5) and also of Commendations (26). In raw
  numbers, quality improvement Recommendations
  outnumber Commendations 21.

15
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
16
Comparison (cont.)

• Whats more, an impressive 74 of the Quality
  category recommendations, either specifically
  mentioned, or it can be reasonably concluded from
  the wording that there is a need for a more
  systematic approach a more consistent deployment
  of the approach across the institution and/or
  more systematic analysis and follow-up of the
  results of quality assurance and feedback
  mechanisms for continuous quality improvement
  purposes. (p.8)

16
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
17
Comparison (cont.)

• .In a nutshell, this review of a fairly
  substantial sample of Recommendations made by
  AUQA in Cycle 1 suggests that there is a need for
  more quality systems thinking by academic
  planners, and more systematic deployment of such
  plans is needed to assist staff to teach more
  effectively, and ultimately for students to learn
  more productively. (p8)

17
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
18
5. Conclusions

• From this study, involving as it does the
  comparison of the pattern of commendatory
  findings against the pattern for recommendations
  for improvement, this author concludes that a
  student-centred approach applies equally, but
  one in which the student evaluation of teaching
  and other feedback systems provide the vital
  information for academic decision making about
  the perceived quality of the student experience.
  (p8)

• From their study of AUQA Commendations, Radloff
  and de la Harpe concluded that there was a need
  to provide more support for intellectual
  development and good curriculum design and saw
  the adoption of a student-centred approach to
  learning with a focus on intellectual development
  as a way of achieving that.

18
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
19
But what can educational / LT development units
do?

• On page 9 of the Paper, there is a list of 27
  qualitative statements taken from the Quality
  improvement findings. Here are just a few
• systematically implement procedures for student
  feedback
• unification of rules or guidelines for
  implementation of University policies
• consistent collection and clearer identification
  of responsibilities
• clearly outlined within a framework of
  accountability for action
• ensure that feedback is given to students on
  results and follow-up
• undertake evaluative studies of student
  performance
• regularly subject to thematic analysis to
  determine systematic improvement

19
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
20
What can educational / LT development units do?
(cont.)

• If there is a unifying theme running through
  all these findings, it has to be that there is a
  great need for more systematic (consistent)
  implementation and a need to delegate appropriate
  responsibilities for follow-up actions. (p9)

20
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
21
What can educational / LT development units do?
(cont.)

• A final observation of this
• It is worth noting that quite a number of the
  LT Commendations explicitly mentioned the good
  work of educational / LT development units,
  while there was only a handful of Recommendations
  for improvement that mention such units. (p9)

21
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
22
6. A Closing Word on AUQAs Cycle 2 Audits

• AUQAs second cycle of audits will commence in
  2008. The focus will be on Closing the loop
  from the Cycle 1 audits and on evaluating
  Academic Outcomes and Standards. It will be
  possible for universities to nominate themes.
  Likely themes for LT are Academic QA
  Curriculum and Assessment Academic QA
  Admissions and Progress Quality of Teaching
  and, Learning Outcomes. Benchmarking will be
  vital. (p10)

22
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y
23
Thank you Any questions, or issues?
23
A u s t r a l i a n U n i v e r s i t i e s Q
u a l i t y A g e n c y