Conducting programme evaluationsreviews with a view to sustainable quality promotion

1
Conducting programme evaluations/reviews with a
view to sustainable quality promotion

• Jan Botha
• University of Stellenbosch
• South Africa
• Keynote Address
• Oman Quality Network Conference
• Muscat, 28 October 2008

2
Key questions

• Can we apply the 80/20 principle to programme
  evaluations?
• What are universities learning from the
  evaluation of (formative) programmes?

3
Outline of the paper

• Suggestions for the academic leader
• QA in higher education context
• Misconceptions about QA in higher education
  context
• Programme evaluations and the 80/20 principle
• What are we learning from programme evaluations

• Broader perspectives Context
• Literature
• Experiences
• Narrower focus on a specific issue
• Empirical
• Analytical

4
Suggestions for the academic leader
5
The predicament of the academic leader

• I am a dean / HoD, my research ?
• I have become one of them
• In the middle of the sandwich
• I am herding cats
• I have to manage quality
• From critic to custodian

6
Academic Leadership

• Learning Leaders in times of change
• Geoff Scott, Hamish Coates, Michelle Anderson
  (May 2008)

7
Suggestions for the Academic Leader

• Use quality assurance as instrument to achieve
  your vision and goals for your School /
  Department / Programme
• Quality assurance can be a powerful change
  mechanism
• Use a research-informed approach to quality
  assurance
• QA? Understanding / Insight ? Action

8
QA as an Academic / Scholarly Project

• QA to be approached as a form of research
• an intellectual enterprise
• driven by curiosity, creativity and pragmatics
• underpinned by methodologically sound inquiry
• accompanied by experimentation and reflection
• informed by theorising
• resulting in action, application and peer
  reviewed publication

9
A Principle of Design
Specifications are continually moving upwards
Opportunity for incremental changes
Specifications
Calculate this cost
Poor design
Poor design
Good design
10
Quality Assurance in Higher Education Context
11
Living with ideology in the university

• Ronald Barnett
• The idea of the university as a site of
  reason
• Universities are beset by many ideologies
• ideology is that which is opposed to truth
• Competition and Entrepreneurialism
• Quality
• Managerialism
• Research
• Access, inclusivity, multiculturalism
• Turn the power of ideology onto itself, move from
  ideologies to ideaologies recall the idea of
  the university

Beyond all reason. Living with ideology in the
university. 2003. Open University Press.
12
The idea of the university?
13
What are universities for?

• Universities state their vision, mission and
  objectives
• A place of quality, a place to grow
• Creating futures
• Your Knowledge Partner
• Innovation Generation
• To advance Learning and perpetuate it to
  Posterity
• Sultan Qaboos University aspires to be an
  academic institution of excellence recognized
  both nationally and internationally, a center for
  learning where individuals can develop their
  capabilities in an environment that promotes
  academic achievement and research excellence and
  a community where the highest moral and ethical
  values prevail for the purpose of both
  self-advancement and service to the community.

14
International snapshot

• International Network for Quality Assurance
  Agencies in Higher Education (INQAAHE) a
  world-wide association of some 200 organisations
• Members include
• Commission for Academic Accreditation (CAA), UAE
• Netherlands Quality Agency (NQA)
• Malaysian Qualifications Agency (MQA)
• Australian Universities Quality Agency (AUQA)
• Comision Nacional de Evaluacion y Acreditacion
  Universitaria. (CONEAU), Argentina
• National Accreditation Agency (NAA), Russian
  Federation
• Oman Accreditation Council (OAC)

15
South Africa

• Council on Higher Education (CHE)
• just hang on, it will go away,
• as has happened with the other acronymous
  bodies
• Institutional Audits
• Programme Accreditation
• National Reviews of Programmes
• Quality Promotion and Capacity Building
• Three steering instruments in Higher Education

16
(Slide Ian Bunting)
17
Misconceptions about Quality Assurance in Higher
Education
18
Challenging the management of quality

• Is managerialism not contradictory to the
  nature of a university ?
• Management of quality used to cover for poor
  quality?
• More management of quality results in less
  quality
• CHE publication
• Academic Freedom, Institutional Autonomy and
  Public Accountability in South African Higher
  Education (August 2008) (www.che.ac.za)

19
Misunderstandings (1)

• Point of departure lecturers are doing poor
  quality work,
• we need QA to catch them out
• I am satisfied with 95 of the work of the people
  in my university/faculty/school
• therefore QA is not necessary
• Just make sure that you meet the minumim
  standards and keep them off your back
• Aim for the minimum
• Comply

20
Misunderstandings (2)

• If good academics are appointed at universities,
  we do not need QA systems. If the wrong people
  are appointed, no document (however thick it is),
  will ensure quality.
• If nobody steals, we do not need laws and police.
  If there are thieves, no law (however thick it
  is), will ensure that no theft takes place.
• If nobody is ill, we do not need doctors. Since
  we know beforehand that everyone is in any case
  going to die some day, the whole medical
  profession is senseless.

21
The case for Quality Management

• Quality management is a necessary condition for
  quality,
• however,
• quality management is not sufficient for quality

22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
Programme evaluations and the 80/20 principle
28
Acknowledgements

• Co-researchers
• Professor Carools Reinecke
• Emeritus Vice-Chancellor, Potchefstroom
  University (South Africa)
• Mr Kamal Bhagwandas Jogibhai
• University of the Witwatersrand, Johannesburg
  (South Africa)
• Research Assistant
• Mr André Müller, University of Stellenbosch

29
The Pareto Principle

• Vilfred Pareto (1848-1923)
• Wealth distribution in 19th Century England is
  predictably unbalanced
• Most income and wealth go to a minority of people
• Consistent mathematical relationship between
  proportion of people and amount of income and
  wealth
• Principle of Imbalance
• Zipfs Principle of Least Effort
• 20-30 of resources accounted for 70-80
  activity
• Jurians Rule of the Vital Few
• the 80/20 Principle

30
The 80/20 Principle illustrated
Effort
Results
R Koch 2007. The 80/20 principle. The secret of
achieving more with less. London N Breadly
Publishing
31
What is a programme?

• A social intervention
• A set of planned, structured learning experiences
  leading to a qualification

32
Purposes and types of programme evaluations

• Evaluation outcomes are used by different role
  players for different purposes
• Mouton Babbie (The practice of social research.
  Oxford, 2001)
• judgement-orientated evaluations
• improvement-orientated evaluations
• knowledge-orientated evaluations
• Trow (Academic reviews and the culture of
  excellence. Stockholm1994)
• internal supportive
• internal evaluative
• external supportive
• external evaluative

33
Unit of evaluation/review

• An institution as a whole
• An academic organizational unit within an
  institution
• department, centre, institute, bureau, school,
  faculty
• Specific activities or processes within an
  institution
• learning and teaching programmes
• research programmes
• community engagement programmes
• capital campaigns

34
Types of programme evaluations

• Evaluation of professional programmes as part of
  the periodic programme accreditation process
• conducted by professional councils
• National reviews of programmes
• conducted by a national QA agency
• the same programme evaluated and (re)accredited
  simultaneously at all institutions
• Internal programme evaluations/reviews (for
  different purposes improvement/closure/consolidat
  ion etc.)
• conducted by institutions
• professional and formative programmes
• undergraduate and postgraduate programmes

35
Outcomes of national reviews in SA

• 37 MBA programmes offered by 18 public and 9
  private higher education institutions evaluated
  during 2003
• 5 fully accredited,
• 15 conditionally accredited
• 19 Master of Education programmes evaluated
  during 2005
• 7 fully accredited,
• 2 provisional,
• 5 conditional,
• 2 postponed,
• 3 de-accredited

36
Challenges in programme evaluation

• Coherence within broader internal and external QA
  system
• Formative programmes as unit of evaluation
• BA, BSc, BComm, B SocSc
• Spread over many departments in Schools and
  Faculties
• Not to conflate process of evaluation with
  reporting the results of an evaluation process
• Too much effort and resources for outcomes? 80/20
• Administrative burden
• Evidence

37
(No Transcript)
38
Expectations of Programme Evaluations

• Sustainable quality promotion
• An instrument for change
• Integrated with other forms of evaluation
• Outcomes should justify the effort
• for academics and evaluators and HE managers
• Standard methodology used
• self-evaluation based on criteria/standards
• Self-report
• panel visit, report
• improvement action, monitoring
• Applicable to formative programmes

39
A Pareto Approach to Programme Evaluation

• Consider the ideal situation
• refer to criteria / standards
• clustered in 11 themes
• simplified (and translated)
• Formulate a key question (or questions)
• focus, contextualise (what does this mean for us)
• Express the evaluation in a symbol (5 point
  scale) discipline to make a judgement
• Motivate the evaluation symbol (brief narrative,
  with substantiating evidence)
• Formulate strategies (in bullet points)

40
Template for the Self-evaluation Report

• Programme rationale
• Criteria
• The programme is consonant with the facultys
  mission, planning and resource allocation. The
  design maintains an appropriate balance of
  theoretical, practical and experiential knowledge
  and skills. It has sufficient disciplinary
  content and theoretical depth at the appropriate
  level.The programme offers opportunities for
  community interaction.The design offers learning
  and career pathways to students with
  opportunities for articulation with other
  programmes within and across institutions, where
  possible. The existing formulation of the
  strategic importance of the programme was
  re-evaluated.
• 1.2 Key Question
• 1.3 Evaluation
• 1.4 Motivation
• 1.5 Strategy
•  

Reports limited (8,9 pages). Strategies
formulated as actions
41
Criteria/standards clustered in themes

• Programme rationale
• Academic integrity
• Student recruitment, admission and selection
• Staff
• Learning facilitation
• Assessment
• Infrastructure and academic information sources
• Programme co-ordination
• Student success and academic support for student
  success
• Service Learning and work-based learning
• Programme Evaluation and Development
• Research basis (postgraduate programmes)

42
What are we learning from programme evaluations?
43
Case study

• Faculty of Sciences, Stellenbosch University
• Bachelor of Science in Physics Chemistry,
  Mathematical Sciences, Earth Science,
  Biodiversity, Molecular Biology, Human Life
  Sciences, Sport Science, Science Education
• Faculty of Agri-Sciences, Stellenbosch University
• Bachelor of Science in Agricultural Sciences in
  Animal Production Systems Agricultural
  Economics Wine Production Systems Crop
  Production Systems Forestry Food Science
• Faculty of Humanities, University of the
  Witwatersrand (Johannesburg)

44
Agri-Sciences
45
Agri-Sciences
46
Sciences
47
Sciences
48
(No Transcript)
49
Concerns, mild concerns, commendations

• Serious concerns
• Recruiting the right students, adequately
  prepared widening participation (Omanisation),
  optimal number
• (Agricultural Sciences and Natural Sciences)
• Work-based learning
• (Agricultural Sciences and Natural Sciences)
• Concerns
• Programme rationale
• (Agricultural Sciences)
• Infrastructure, facilities, equipment
• (Agricultural Sciences
• Commendable
• Infrastructure, facilities, equipment (Natural
  Sciences)
• Learning facilitation (Agricultural Sciences)
• Assessment (Agricultural Sciences)

50
Strategies for Improvement(Selected and
complied from 574 strategies for improvement
listed in the self-evaluation reports)
51
1. Programme rationale (69)

• To enhance interaction with stakeholders (subject
  specific societies, industry, extraordinary
  lecturers, alumni) to broaden academic and
  industry specific networks (through a voluntary
  advisory committee)
•  To review/restructure the focus and content of
  the programme annually to ensure that module
  outcomes strongly reflect the programme specific
  and generic outcomes, reviewing programme
  structures annually and implementing student
  feedback and industry input, and developing a
  formal system to gather feedback from graduates
  on their appraisal of the course, and suggestions
  for improvement.
•  To adapt and better integrate courses, share
  expertise/reduce duplication and use resources
  more cost-effectively, to reduce lecturing load
  by constructing generic courses across
  departments and including and designing modules
  to fill theoretical gaps and deepen theoretical
  knowledge or to add attractive career paths
•  To define service/experiential/work based
  learning within the programme and incorporate
  electives/optional subjects to prepare students
  better for the work environment, extending
  practical training/internship periods and
  awarding credits for lab or field work
  experience To offer more opportunities for
  community interaction, especially for graduate
  students.

52
1. Programme rationale (continued)

•  To communicate outcomes clearly to students to
  contextualize lectures and learning experiences
  to communicate information about administrative
  and support services to students, staff and
  stakeholders (advertising student assistantships
  effectively)
•  To identify and support students at risk in
  small groups and give academic and non-academic
  help and to communicate the importance of class
  attendance also addressing low/increased student
  numbers
•  To sharpen recruitment and marketing efforts,
  improving outreaches where scholars, prospective
  and current students can be informed about the
  programme and its strategic value. (Open days,
  information sessions...)

53
2. Academic integrity (65)

• To increase reseach and benchmarking
  opportunities with international scholars, to
  ensure the programme remains at the forefront of
  new developments, and to encourage
  research/teaching interaction
•  To review the undergraduate programmes annually,
  continuousand coherently planning of courses in
  terms of content, level of difficulty and credit
  value constantly reviewing the curriculum,
  learning materials, learning methods and
  programme outcomes moderating it and ensuring
  that the feedback from external moderators is
  taken into account
•  To increase collaboration between lecturers to
  improve programme cohesion, filling in
  theoretical gaps through new and adapted courses,
  and to cut out duplication also working with
  timetabling administrators to free up course
  combinations to provide better articulation with
  cognate programmes, to structure the programme to
  expose students to the core themes, to balance
  practice and theory better to ensure the
  relevance of prescribed courses that are
  presented by other departments from both within,
  and external to, the school/faculty
•  To identify and remove obstacles inhibiting the
  use of experential learning and increase
  laboratory time, re-evaluating the module
  composition and structuring of the programme in
  order to make place for a longer period of
  internship to evaluate the efficacy of the
  practical portions of the modules and investigate
  coherent year-long practical modules at second
  and third year levels
•  To communicate the reason for the approach
  followed during the first year of study, and
  maintain a challenging learning environment for
  students despite low/increased numbers

54
2. Academic integrity (continued)

• To develop generic skills (writing, information
  and computer) and improve the teaching of
  practical skills through the use of continuous
  assessment for all modules from second year
  upwards, and to identify and adapt modules fairly
  early in the academic programme To focus on
  improving student mathematic skills To complete
  computer literacy and scientific communication
  skills modules in the first year
• To manage or establish new research
  institutes/units/centres such as an Institute for
  Food Technology (IFT) or Institute of Food
  Science and Technology (IFST)
• To recruit students from historically
  disadvantaged backgrounds
• To recruit more international students

55
3. Student recruitment, admission and selection
(93)

• To involve industry stakeholders in recruitment
  and funding for undergraduate scholarships To
  acquire additional bursaries from all sector role
  players To expand facilities and staff To
  increase support staff, especially in terms of
  popular media and marketing To involve subject
  societies in recruitment actions To increase
  interaction with bursary providers about the
  criteria for successful students and the problems
  that are experienced by current bursary holders
  To investigate the roles professional bodies play
  in promoting the image of certain careers
•  To review admission requirements for mathematics
  and science related to forest science programme
  To evaluate the tempo at which graduates are
  employed To investigate existing recruitment
  actions and material in terms of success and
  relevance To monitor the extended degree
  programme to ascertain whether it is increasing
  student diversity To monitor student numbers at
  the beginning of the year and consider the
  adjustment of entrance criteria if necessary To
  review policy on ranking on application form so
  as to admit students who desire to follow To
  review academic support available to struggling
  students in all years
•  To adapt intake in terms of quantity, quality
  and equity of students. To align recruitment
  efforts of the university, faculty,
  programme-committee and departments To develop
  and implement mechanisms, including assessment
  methods, to widen access, (e.g. summer school,
  bridging programme) To increase the diversity of
  the student body in terms of South African
  cultural groups and international students To
  create a structured marketing and recruitment
  plan as top priority, buying in expertise if
  needed, consulting with SU marketing divisions,
  utilizing existing marketing mechanisms,
  negotiating for undergraduate bursaries and
  addressing the student diversity To improve the
  administrative implementation of admission
  criteria To reconsider admission requirements at
  first year level to curb the high failure rate,
  as well as at honours level (possibly) To
  re-assess the modules which have become service
  courses

56
3. Recruitment, admission and selection
(continued)

• To make the bridging programme compulsory for
  students with a Gr 12 mark between 50-56 To
  visit underprivileged schools in our immediate
  vicinity To sponsor prizes (e.g. book prizes)
  for the best Biology (now Life Sciences?) student
  in Grade 12 at a few selected schools To
  consider an orientation period including visits
  to departments as well as the experimental farm
  before registration of second-year students to
  allow informed choices on major subjects To
  supply information on programmes at the Expo for
  Young Scientists and Olympiad candidates, as well
  as high school science teachers To encourage
  third years to attend final years product
  development presentations To focus on the
  recruitment of coloured students due to the
  demand from Agri businesses
•  To implement extended degree programmes (and
  first year academy) to benefit students that have
  to overcome academic inadequacies due to
  historical barriers, and help students to
  overcome cultural and language difficultiesTo
  prevent over-subscription to the course Cap
  student numbers (no more laboratory space
  available). To identify the optimum number of
  students that can be accommodated in the
  programme (estimated at between 40 and 50
  students), keeping economic factors in mind To
  market a BSc(Hons) as a career route both
  commercially and in parastatal, particularly
  academic spheres To pay careful attention to
  entrance requirements in the next few years as
  the new school leaving certificate is
  implemented. Aim to raise entrance requirements
  To critically review failures and examine the
  reasons for these
•  To initiate discussions with the government on
  the fragmentation of higher forestry education in
  South Africa To broaden the communication base
  with students at first- and second year level To
  develop a culture of uniqueness and excellence
  amongst staff members and students To raise
  awareness of language issues in the academic
  programme

57
3. Recruitment, admission and selection
(continued)

• To make better use of recruitment opportunities
  (e.g. Open Day, visits to schools) To recruit
  black students via industry contacts To send
  promotional material (pamphlets) to all Secondary
  schools in the Western Cape To represent the
  department at open days organized by IPS To
  write an article promoting Conservation Ecology
  and Entomology in general and our department in
  particular for the popular science magazine
  Quest To ensure the web-site inspires
  students To continue with current promotions
  activities and transparent admission policies To
  have greater flexibility within the language
  policy To provide admission requirements into
  the program (especially as of 2009) on
  departmental and faculty web pages and
  brochures. To advertise the programme campus
  wide on notice board, emphasizing the financial
  support through industry bursaries, especially to
  students from disadvantaged backgrounds. To
  promote the need for a Faculty Open Day with a
  smaller, but more specific learner group (eg. top
  10 learners within a grade with mathematics as
  school subject or learners from strong feeder
  schools) so that departments can participate more
  effectively. Our perception is that the
  University Open Day fails to recruit quality
  motivated students for the program. To improve
  the distribution of marketing material To
  recruit talented learners with bursaries To
  contact CPS to disseminate the information on
  courses in the programme, job opportunities and
  available bursaries To give more students access
  to bridging programmes To try to attract the
  best Afrikaans and English students To attract
  black students with the Fundza Lushaka bursary
  scheme

58
4. Staffing (62)

• To enhance opportunities for continuous
  professional development To make sure all newly
  appointed lecturers attend the Universitys
  induction and training courses To ensure
  on-going professional development, letting staff
  attend various courses (e.g Web-CT training) as
  the need arises. To deliver graduates who are in
  demand in the food and cognate industries To
  develop C2 and C3 staff new skills and knowledge
  To enable personnel to develop academically in
  their fields of specialization To encourage
  research activity by giving all staff members an
  off semester with no undergraduate teaching
  commitments and additional training in assessment
  methods for large classes To maintain good
  research profiles and to expose students and
  lecturers to current research and teaching issues
  and trends To quantify time usage and check
  against workload document to enforce contract
  hours To manage time more effectively by setting
  fixed consultation times
•  To beter utilize the annual performance
  evaluation process, Permanent staff members to be
  required to compile a personal work agreement and
  performance contract. This assists in quality
  assurance of teaching and learning, research and
  service To have all teaching staff, with one
  exception, have a PhD To review teaching load of
  staff through programme evaluation To have
  positive departmental evaluations To ensure an
  equitable distribution of the teaching load by a
  yearly review To review and evaluate the
  deployment of technical staff for optimum
  technical support
•  To ensure more realistic work loads for academic
  staff To align specialisation and research focus
  of staff needs with programme. To ensure greater
  cooperation and less duplication

59
4. Staffing (continued)

• To utilize personnel seconded from industry to
  the departments can assist in maintaining
  research capacity To investigate mechanisms by
  which specialised technical / practical knowledge
  can be imparted to students. This may be linked
  to a specific academic staff member, specific
  tasks during the internship, and/or exposure
  during practicals To ensure that student
  assistants are well-trained
•  To better communicate the benefits of an
  excellent diversity profile and the spin-offs for
  the student body, who are exposed to a new
  experiences, approaches and ideas. To inform the
  faculty through the departmental evaluations that
  a critical mass of academic staff and sufficient
  administrative support are not present within all
  departments in the program to effectively perform
  and excel in all 3 areas expected namely
  teaching, research and community service To
  actively encourage students and lecturing staff
  to interact about the expectations of web
  presence of modules, particularly in smaller
  senior modules
•  To maintain standards and ensure quality despite
  the increase in student numbers To manage
  part-time staff carefully To have adequate
  facilities and personnel to allow students to do
  practical work and experiments
•  To address the perceived insufficient levels of
  remuneration to attract and retain young
  academics, especially for young black staff. The
  current personnel evaluation has shortcomings and
  financial incentives linked to excellence are
  insufficient to motivate academic staff.

60
5. Learning facilitation (70)

• To get more clarity on the meaning of student
  centered teaching and its implications To
  encourage participation by academic staff in the
  activities of Centre for Teaching and Learning
  To utilize WebCT to effectively communicate with
  large groups, but not replace lecturer-student
  interaction or class notes To evaluate and
  compare learning material with international
  universities To develop a policy on student
  centered teaching so that independent,
  enthusiastic and spontaneous learning
  consistently takes place To incorporate
  fundamental knowledge much more overtly
•  To act on students feedback and evaluation To
  give a copy of each module evaluation to the
  programme coordinator (improve efficiency of the
  feedback) To request lecturers to complete
  module frameworks, literature bibliographies,
  goals, outcomes in accordance with the module
  framework requirements of Senate To review the
  relevance of all units To revisit the PBL
  process in terms of life long learning, critical
  thinking and professional reasoning To review
  what is lectured, the links between problems and
  lectures and workshops and feedback To review
  problems and evaluate their incremental demands
  over the four years To review decision to make
  curriculum books only available to students in
  e-format
•  To align all study guides with the Universitys
  policy requirements for study guides
  (specifically the inclusion of module-level
  outcomes in study guides) Investigate ways to
  make class experience more stimulating To make
  more use of text books and journal publications,
  and less class notes Powerpoint slides always to
  be provided to students To employ a variety of
  assessment opportunities to evaluate the
  facilitation of learning To use text books and
  scientific publications to a greater extent and
  in preference to class notes To review the
  modules to ensure that they contain learning
  opportunities for the development of these
  skills, without unnecessary duplication To
  relook at the orientation course in first year
  (preparation for PBL curriculum) and add a test
  to ensure the content is mastered and
  internalized

61
5. Learning facilitation (continued)

• To investigate the option of providing credits
  for laboratory or field work experience
•  To adequately communicate the module outcomes to
  the students yearly by the chairperson and via
  the website To advertise student assistantships
  opportunities effectively To expose second
  year students to the layout and cohesion of the
  programme To involve students in more personal
  feedback (similar to this evaluation exercise) at
  a module and programme level To highlight the
  fact that ability to work in a team is part of
  the supposed programme outcomes and establish
  where this is in fact done
•  To assign tutors dedicated to the support of
  disadvantaged students To inform departments of
  the need to apply computer literacy skills and
  gradually introduce sourcing and use of
  scientific literature earlier in the program
  (3rd-year level). Basically, do not leave all
  tasks, seminars and research projects for the 4th
  year To discuss with computer literacy convenors
  options to allow mathematical science students to
  do fewer but more relevant modules within
  computer literacy To investigate the possibility
  of introducing opportunities for students to
  improve and perfect their written and verbal
  communication skills at early stages in their
  studies To develop oral presentation skills for
  senior students

62
6. Assessment (58)

• To ensure that all staff members are aware of
  departmental policies and requirements for
  assessment and the publication and storage of
  marks To use peer-reviewing within student/study
  groups as an effective supplementary method of
  assessment To encourage continues assessment
  training of staff To change fieldwork rubrics to
  be more user-friendly and precise (with student
  input) To review the number of assessment
  activities that contribute to the marks and
  activitiesTo review time needed to answer
  questions and consider the advisability of seen
  vs unseen questions To analyze all exam
  questions according to Blooms taxonomy
•  To align assessment practices all course
  frameworks, study guides with the universitys
  requirements To ensure that all tests and exams
  are aligned with the principles of the
  Universitys Assessment Policy To investigate
  Turn-It-In for electronic submission of
  assignments To investigate ways to reduce the
  significant increase in administrative load
  associated with internal and external moderation
  To have rigorous internal moderation, and
  external moderation for third year and honours
  modules To establish, manage and oversee an
  accurate and reliable administrative system to
  record marks onto the SIS.
•  To enhance strategies to eradicate plagiarism
  To make assessment challenging, in particular to
  test problem-solving abilities To give more
  smaller tests rather than only a few major tests
  and an exam To assess individuals fairly in a
  group assignment and CTL should be approached for
  assistance in this regard To pursue a more
  systematic manner for collecting student feedback
  on each module To improve module frameworks to
  include assessments details (dates, type of
  assessment as well as expected timeframe for
  feedback) To use a range of assessment methods
  such as a seminar, laboratory, written and oral
  examinations

63
6. Assessment (continued)

• To clearly communicate the means by which
  problem-solving abilities will be assessed, i.e.
  the quality of the questions to be expected, the
  amount of insight that will be required To
  update the assessment dates and weights on the
  website To keep lecturing staff (and yearbooks)
  updated with regulations regarding assessment and
  moderation (internal and external) at
  departmental level To communicate the different
  assessment methods of different modules clearly
  to the students To provide reasons or
  motivations for giving a particular mark,
  especially for essay-type projects and similar
  essay-type exam questions
•  To conduct individual interviews with students
  scoring gt30 in semester test, determine reasons,
  plan for support To devise an early warning
  system for students who are struggling (more
  difficult with larger classes) To give attention
  to entrance requirements and streaming of
  students to preserve standards To handle
  question papers with care to avoid corruption of
  the assessment process
•  To monitor individual student progress in terms
  of the First Year Academys mechanisms.

64
7. Infrastructure and academic information
sources (55)

• To make sure that teaching facilities remain to
  up to date To improve, maintain and replace
  laboratory equipment To challenge the
  insufficient budgets to support the running cost
  of undergraduate practical training in some
  courses (especially those with high student
  numbers) To expand laboratory space, personnel
  To develop IT support services to personnel
•  To use our excellent library services optimally,
  faculty librarian specialists and proper student
  training and regular purchases of new DVDs and
  material To investigate the possibility of using
  the computer users area for training and tests
•  To re-allocate facilities to departments so that
  lecture theatres, laboratories and the cellar can
  be in the same building To arrange for continued
  access to academic scientific information
  (library) once graduates are working in rural
  areas To maintain the effective academic support
  offered by the Library, SunMedia and Information
  Technology. To monitor access to and choice of
  electronic journals To identify one or more
  modules to be enriched in teaching of writing
  skills and use of library facilities and
  undergraduate students should be required to
  submit a literature review at some point in the
  academic programme to ensure that the desire to
  continue reading and learning beyond graduation
  is fostered.

65
7. Infrastructure and academic information
sources (cont)

• To improve the equipment for undergraduate
  practicals so that each student can complete
  practical assignments individually To update the
  computer facilities and associated software To
  optimize laboratory equipment for teaching
  purposes (more student-proof research equipment)
•  To optimally utilize the class space, modern
  equipment, laboratory facilities, and computer
  and library services
•  To maintain laboratory equipment To monitor the
  lecture hall size amount of students
•  To attend to the insufficient field
  laboratories To prevent overshooting capacity of
  a maximum of 40 students in the programme To
  challenge the tariffs quoted by the General
  Maintenance Section of the University for
  services are perceived to be inflated due to a
  sub-contracting system and is not seen to be
  consistently offering value and high quality
  service.

66
8. Programme coordination (61)

• To integrate programme inputs from different
  stakeholders into curriculum development
  undergraduate, post-graduate, extraordinary
  appointments, alumni and industry
• To align teaching practices with international
  best practrice To state programme outcomes
  (generic and specific) for all lecturers and
  students involved in the programme through
  updated module frameworks
•  To monitor/review service levels regularly on
  all levels for relevancy within the programme To
  evaluate and implement final year feedback To
  monitor throughput of modules and support
  modules To monitor student feedback earlier in
  the modules to report back To consider student
  representation in the Programme Committee
•  To institute ongoing actions to improve and
  evaluate the efficiency and quality of the
  teaching programme To align individual courses
  and course materials towards the main focus of
  the programme, especially for new members of
  staff To improve communication between
  departments to avoid unnecessary overlap between
  courses and course material

67
8. Programme coordination (contunued)

• To ensure better communication between lecturers
  in different departments about the contents of
  programme To introduce programme coordinator to
  1st year students provide contact details To
  ensure regular communication between teaching
  administration at the University, and teaching
  staff and students To introduce the programme
  co-ordinator to first and second year students,
  and provide contact details in class and on the
  internet To ensure that module frameworks exist
  for all modules, also on the website To hold
  more frequent committee meetings and to create
  more regular opportunities for student input and
  participation To explain lines of communication
  and authority to students
•  To identify and support at risk students by
  tracking their records and analyzing it to help
  the department To explore how the information in
  the Student Oracle system can be utilized To
  ensure that deadlines are met
• To gain better understanding of the reasons why
  certain students do not study successfully

68
9. Student success and academic support (60)

•  To provide more opportunities for individual
  face to face contact between lecturers and
  students To develop presentation skills for
  lecturers and technical staff To engages with
  CHSE to ensure academic staff have adequate
  educational background and facilitation and
  assessment skills To give both new and
  experienced staff the opportunity to attend CTLD
  assessment and teaching courses To adhere to
  University Teaching Assessment policy To
  maintain classes on/below 40 students per class
•  To improve the monitoring system for 1st and 2nd
  year students, To monitor student success more
  systematically To review mentorship strategy
  with a view of developing a formal approach,
  close contact student / lecturer To monitor
  continuously that the pressure to improve the
  pass rate of students, especially at first year
  level, is not carried over on to the second- and
  third year level to finally compromise the
  integrity of the whole program To monitor the
  success rate in every module in the department
  (and faculty) where it resides.
•  To adapt assessment techniques to maintain good
  academic standards To make the existing
  mentoring program more accessible to students in
  the programme. To evaluate and adjust the
  mentoring system in order to make it more
  effective and to ensure continuity from 1st year
  onwards. To look afresh at entrance requirements
  and options for students who struggle To
  investigate the possibility of continuing the
  mentor system for support of students at second
  and third year levels, in addition to the current
  first year support.

69
9. Student success and academic support
(continued)

• Improve support to part time/sessional staff in
  their teaching role To teach students to improve
  skills rather than just to supervise and critique
  them
•  To assist students to overcome the challenge
  posed by language. Dubble medium options for
  courses could be made more readily available To
  remind lecturing staff that all assessments,
  including small tutorial tests, should be
  bilingual To encourage language use accessible
  to all students
•  To alert students to the available student
  support services To scrutinise studentsstudy
  records in all their modules To improve
  language, numerical and cognitive skills within
  the programme To let students participate in
  the first year Learning Academy To assist
  students with the challenges of the first-year of
  the program. The Extended degree program (also
  First Year Academy) is seen as an option to
  improve student success To make better use of
  tutor demonstrators. To extend the successful
  tutor system to other modules if financial
  support can be obtained from the Faculty To
  identify at risk students earlier and offer
  compulsory extra tutorial time To identify good
  role models for students
•  To provide much broader access to an extended
  degree, especially within the mathematical
  sciences. Come to terms with the fact that a
  4-year degree is the norm and use this to plan
  curricula. Consider a universal 4-year programme
  with some students emerging with an honours
  others with a BSc.

70
10. Service learning and work-based learning (43)

• To cooperate with stakeholders to create
  opportunities for WBL interaction with industry
  for all students To update lists of industry
  knowledge partners To explore collaboration
  efforts with MoUs to clarify roles To ensure
  that the commercial entity is suitable for
  receiving undergraduate students and that there
  is no family connection with the student
•  To review and improve the WBL coordination
  responsibility guidelines and monitor the success
  and/or need for WBL through departmental
  evaluations To follow up past-graduates to see
  how feasible their recommendations are
•  To adapt programmes to integrate WBL modules To
  use experimental farms optimally To lengthen
  period of (vacation) internships To extend
  mentoring system to alumni in the industry
•  To ensure the commercial entity is fully aware
  of its WBL obligations to the student before
  placement (e.g. internship) To require
  literature searches during internships To
  formalize community interaction projects with
  SUSPI To consider the possibility of site visits
  and voluntary internships
•  

71
10. Service learning and work-based learning
(cont)

• To communicate to all parties the outcomes of the
  WBL and what the student should be achieving, To
  ask all module coordinators to discuss work
  opportunities in the context of their modules
•  To monitor WBL risks continuously e.g. to
  provide mentoring, support and networking options
  for supervisors (employers) To overcome logistic
  difficulties of WBL in certain industries, e.g.
  the mining and petroleum industries in the
  Western Cape
•  To encourage students to take initiative for
  finding WBL placement
• To address any WBL misconceptions in the industry
  regarding the abilities of graduates and the
  balance of theoretical, practical and
  experiential teaching and training during the
  academic programme. To challenge lecturing staff
  to guard against modules being too theoretical
  given the context in which they are taught. To do
  an impact study to measure the effectiveness of
  the programme in preaparing community servers for
  practice

72
11. Programme evalution and coordination (44)

• To maintain regular interaction with sector
  stakeholders and survey employers on the impact
  of the programme and levels of satisfaction To
  improve liaison with industry / employers To
  improve industrys understanding of the purposes
  and outcomes of the programme To review the
  mechanism used to give stakeholders information
  about curriculum changes based on their input
•  To implement coordinated strategies to review
  programme content and outcomes annually To
  obtain data on student recruitment generated
  during departmental evaluations To ensure that
  all departments obtain feedback from industries
  To send evaluation forms to alumni one year after
  graduation To evaluate all quality assurance
  practices used in the department
•  To adapt courses to eliminate redundancy and
  streamline the learning process To evaluate the
  relevance and efficacy of all modules, including
  the practical portions thereof
•  To initiate a process of communication and
  interaction with departments at this university
  and at other institutions to ensure that we are
  offering modules of the highest standards and
  relevance also consider the employability of
  exit level students
•  To encourage final year students to continue
  with post-graduate studies to better their
  qualifications and improve their employability
  and develop a society for past graduates

73
So what are we learning from programme
evaluations?

• Remember Scientists, Formative, Undergraduate
• Commitment to maintain high academic standards
• Deep concern about students
• Quality of students (school system) Student
  numbers
• Access and Success
• Not clear about student centered teaching and
  its implications
• Need to communicate clearly to students
• programme and course outcomes
• Programme coherence and rationale
• Need to cooperate and communicate with lecturers
  in other departments (Break through academic and
  organisational silos)
• Improve liaison and cooperation with the world
  of work

74
Reflections and conclusions

• Is the selectivity justified (due to 80/20
  approach)?
• The 20 doctor all students must achieve all
  outcomes
• Each programme must meet all the
  criteria/standards
• Sampling and focus justified during evaluations?
• How to select evidence?
• Just right and just enough

75
Reflections and conclusions

• Different strategies at different levels
  (Programme, Department, Faculty, Institution,
  Higher Education System, and beyond)
• Encourage creativity, research, reflection
• Encourage debate, discussion across silos
• Closing the loop remains one of the biggest
  challenges
• Good systems, procedures
• Good people (at all levels)