Praxis or Poiesis? On the Assessment of Applied and Practice-Based Research

1
Praxis or Poiesis? On the Assessment of Applied
and Practice-Based Research
Researching PraxisPedagogy Culture and Society
seminarGothenburg University Saturday 13th
September

• Dr Alis Oancea
• University of Oxford

2

1. What is applied research?
2. How may applied and practice-based research be
   assessed?
3. How can we begin to describe good applied
   research and its relationship to practice and
   policy? ? three domains of excellence
4. Are the three domains compatible? Is it
   justifiable to expect that they can all offer
   legitimate guidance to assessment?

3
1. What are applied and practice-based research?
4
Demonstrable knowledge

• (37) SCIENTIFICALLY BASED RESEARCH- The term
  scientifically based research'
• (A) means research that involves the application
  of rigorous, systematic, and objective procedures
  to obtain reliable and valid knowledge relevant
  to education activities and programs and
• (B) includes research that
• (i) employs systematic, empirical methods that
  draw on observation or experiment
• (ii) involves rigorous data analyses that are
  adequate to test the stated hypotheses and
  justify the general conclusions drawn
• (iii) relies on measurements or observational
  methods that provide reliable and valid data
  across evaluators and observers, across
  multiple measurements and observations, and
  across studies by the same or different
  investigators
• (iv) is evaluated using experimental or
  quasi-experimental designs in which
  individuals, entities, programs, or activities
  are assigned to different conditions and with
  appropriate controls to evaluate the effects
  of the condition of interest, with a preference
  for random- assignment experiments, or other
  designs to the extent that those designs
  contain within-condition or across-condition
  controls
• (v) ensures that experimental studies are
  presented in sufficient detail and clarity to
  allow for replication or, at a minimum, offer the
  opportunity to build systematically on their
  findings and
• (vi) has been accepted by a peer-reviewed
  journal or approved by a panel of independent
  experts through a comparably rigorous,
  objective, and scientific review.
• No Child Left Behind 8 Jan 2002 (US)

5
(No Transcript)
6
Technical knowledge
Considerations of use? Considerations of use?
No Yes
Quest for fundamental understanding? Yes Pure Basic (Bohr) Use- inspired Basic (Pasteur)
Quest for fundamental understanding? No Pure Applied (Edison)
OECD 2002a 78 original investigation undertaken
in order to acquire new knowledge.., directed
towards a specific practical aim or objective
Stokes, 1997
7

• an area situated between academia-led theoretical
  inquiry and research-informed practice, and
  consisting of a multitude of models of research
  explicitly conducted in, with, and/or for
  practice.

8

• not to disregard reason and principle, and yet
  to attend to them only within the context of an
  even closer attention to concrete situations
  (Dunne, 1993)

9

• therefore we ought to attend to the
  undemonstrated sayings and opinions of
  experienced and older people or of people of
  practical wisdom no less than to demonstrations
  for because experience had given them an eye they
  see aright (Aristotle, EN, 1143b, 10-15)

10
2. How is applied and practice-based research
assessed?
11
Current climate the RAE

• Where researchers in higher education have
  undertaken applied and practice-based research
  that they consider to have achieved due standards
  of excellence, they should be able to submit it
  to the RAE in the expectation that it will be
  assessed fairly, against appropriate criteria UK
  Funding Bodies (2004, para 47)
• RAE subject panels to
• define appropriate criteria for identifying
  excellence in different forms of research
  endeavour, while attaching no greater weight to
  one form over another and () to make provision
  to recognise the diversity of evidence for
  excellent research (UK Funding Bodies 2005, para
  16).

12
Current climate the REF

• The RAE, despite ostensibly supporting applied
  research, may have contributed to the
  reinforcement, rather than the solution, of these
  problems (see criticisms in the Roberts and the
  Lambert reports of 2003)
• Is the new REF likely to solve this problem?
• Scale and funding
• Citation patterns

13
The 2005 project not another set of criteria, but

• (a) recognize the diversity of perspectives on
  applied and practice-based research quality in
  education, and of the ways in which they build
  their legitimacy
• (b) preserve the importance of methodological and
  theoretical soundness in applied, as well as in
  curiosity-driven, research
• (c) emphasize the principle that education
  research ought to be assessed in the light of
  what it wants and claims to be, and not through a
  rigid set of universal standards.

14

• Is research evaluation really a matter of getting
  the technologies right (i.e. of making them
  effective, controllable, and in perfect fit with
  their object)?
• What are the dangers entailed by the focus on the
  fine-tuning of techniques and criteria to objects
  (and the reverse), rather than on the limits of
  the discourse from which these techniques emerge?
• Can we help unsettle current discourses and shift
  the overall way of conceiving of research
  assessment and of research quality from a
  narrowly defined technical concern towards a
  wider (historically and philosophically)
  understanding?

15
Discursive alternatives to instrumental accounts
of research assessment
Alternative discourses ? Complex entanglement of
research and practice and different modes of
knowledge ?Nurturing excellence/ virtue
(epistemic, technical, and phronetic) ?Deliberatio
n and judgment ?Assessment techniques help
research communities to gain increased control
over the contingencies of their practice

• Current discourse
• Hierarchical relationship between modes of
  research
• ?Quality assurance and quality assessment
• ?Quantification, measurement and ranking of
  performance
• ?Assessment techniques unquestioningly produce
  externally-specified outputs

16
3. How can we begin to describe good applied
research and its relationship to practice and
policy?
17

• domains of knowledge (or of engagement with the
  world theoresis (contemplation) poiesis
  (production) praxis (virtuous action in the
  public space)
• excellence or virtue
• episteme theoretike (knowledge that is
  demonstrable through valid reasoning)
• techne (technical skill, or a trained ability for
  rational production)
• phronesis (practical wisdom, or the capacity or
  predisposition to act truthfully and with reason
  in matters of deliberation, thus with a strong
  ethical component)
• (Aristotle, EN, 1139a, 2728, 1178b, 2022).

18
Epistemedemonstrable knowledge

• Application as gradual gliding from the general
  to the particular, and from the abstract to the
  concrete.
• Not the exclusive domain of basic research
• Methodological concerns

19
Techne - technical skill

• using research-based knowledge to attain
  specific, externally defined, ends or to bring
  about certain states of affairs, by controlling
  the resources and procedures involved not
  necessarily instrumental
• Control, systematicity, precision, explanation
• Universality and explanation yield control over
  the future in virtue of their orderly grasp of
  the past teaching enables past work to yield
  future progress precision yields consistent
  accuracy, the minimization of failure (Nussbaum,
  1986, pp. 97)

20
Phronesis - practical wisdom

• First-person action on internal principles of
  excellence
• conduct in a public space with others in which a
  person, without ulterior purpose and with a view
  to no object detachable from himself, acts in
  such a way as to realise excellences that he has
  come to appreciate in his community as
  constitutive of a worthwhile way of life (Dunne,
  1993, p. 10)
• Ethical deliberation about ends and reflective
  choice
• in art, he who errs willingly is preferable, but
  in practical wisdom, as in the virtues, he is the
  reverse (Aristotle, EN, 1140b, 20-25)

21
Paradigm-dependant criteria Transparency/
explicitness Propriety Trustworthiness Advancement
of knowledge
Plausibility Engagement Reflexivity, deliberation
and criticism Receptiveness Transformation,
personal growth
Quality domains
Domains
Marketability and competitiveness Cost Auditabilit
y Feasibility Added value/ brand
Fitness to purpose Salience/ timeliness Specificit
y and accessibility Concern for enabling
impact Flexibility and operationalisability
22
4. Are the three domains compatible? Is it
justifiable to expect that they can all offer
legitimate guidance to assessment?
23

• Generalisable knowledge and teachability
• Ethics and educational transformation
• Experience and professionalism

?
24
Questions

• What concept of the relationship
  research-practice/policy underpins your project?
  
• Which of the domains of the framework (or others)
  does your work foreground? What does this entail
  in relation to the understanding of "good"
  research built into your work?
• Which of the domains of the framework would least
  match your research?