Institutional Research What problems are we trying to solve

1
Institutional ResearchWhat problems are
wetrying to solve?

• Mantz Yorke Bernard Longden

2
Sorting out types of problems
Reports that say that something hasn't happened
are always interesting to me, because as we know,
there are known knowns there are things we know
we know.
We also know there are known unknowns that is
to say we know there are some things we do not
know.
But there are also unknown unknowns the ones we
don't know we don't know.
Donald H. Rumsfeld, Department of Defence - news
briefing, February 12, 2002
3
The challenge facing any University is to Be
smarter and better manage the known knowns
Inquire into university life to seek out the
known unknownsand Reduce the risk of unknown
unknowns as they relate to
Demographic changes
National policy drivers
International opportunities
4
Types of problem Types of institutional research
Two broad types of research
Why is it going on type of research
Whats going on type of research
Explanatory
Descriptive
Without theory, research is impossibly narrow.
Without research, theory is mere armchair
contemplation. (Silverman, 2001 294)
5
Institutional research - four faces
Audience
Summative external Accountability
Formative internalfor improvement
Information
Presentation
Administrative, Institutional
Role
Research forimprovement
Educationalresearch
Academic, Professional
Volkwein (1999) has provided a useful analysis of
the aims of IR The four Faces of IR.
6
Terenzinis organisational intelligence
Hierarchy of complexity
Culture knowledge of what a college is and
where it has come from
Major issues or decision areas facing the
university
Factual knowledgeanalytical skills
7
What problems are we trying to solve?The view
from inside - BL
Audience
Summative external Accountability
Formative internalfor improvement
Information
Presentation
Administrative, Institutional
Role
Volkwein (1999) has provided a useful analysis of
the aims of IR The four Faces of IR.
8
Administrative/Formative-known knowns
IR as Information Authority
Administrative formative and internal
Emerging within institutions as Information
office Management information Planning and
development Institutional Research
Primary functions
Clarifying the known knowns
Describing the university in data terms
Educating the university community about itself
9
What problems are we trying to solve?
Information authority
Base data about the institution profile and
fact book Statutory returns Validating data is
this really us? Size and shape of the institution
to assist in setting targets and plans
Terenzinis organisational intelligence level
Technical/analytical intelligence
In the UK these are well developed possibly over
load the university with data
The challenge is to reduce the plethora of tables
and cross tabs to identify Key Questions that
are crucial to the organisation for success
10
What problems are we trying to solve?
Presentation facility
Marketing the best aspects advocate role Audit
and external accountability Performance
measure League tables Terenzinis organisational
intelligence level Technical analytical
intelligence
Primary functions
Conform data to agreed descriptors
Performance Indicators
Satisfy public funding legislative requirements
Risks and concerns
Spin and managing data
Political pressure to maximise impact of data
Denial about the message data gives
11
Student Data flow
Data informs funding- audit
Student data
Students
Aggregate early return to HEFCE
Data passed to HEFCEwho recreate
Decemberaggregate return
IndividualSummative activitydata return to HESA
But same data also used to inform public
throughPublications, commodification of data
Rankings
12
Data sources to assist with the questions
UK
Higher Education Statistics Agency Published
data sets HEDI
UCAS Unistat
HEFCE (?) National Student Survey
US
National Centre for Education Statistics IPEDS
universe institutional census NPSAS National
Post Secondary Student aid Study BPS Beginning
Post Secondary - Longitudinal Study BB
Baccalaureate Beyond - Longitudinal Study
National Science Foundation
13
Some propositions that we might need to address
Do universities make good use of the data within
their own university?
Does IR constitute value for money?
Is data critical to teachers, administrators,
decision makers?
Is the emphasis between data collection and
analysis appropriate?
Should the IR questions be articulated more
clearly and data collected in response, or should
we continue to collect data and then use what we
have for policy decision?
What lessons can be learned from the research
communitys use of data sets in the US?
14
Comparison sometimes interesting

• UK HESA and US IPEDS similar objectives and
  operation
• US interest in scientific advancement not
  reflected in the UK
• Possibly RAE provides nearest comparison
• US positively encourages Institutional Research
• Association for Institutional Research (US) in
  conjunction with National Centre for Education
  Statistics provides funded summer school to
  encourage analyses of public data
• UK implicitly discourages Institutional Research
  
• HESA charge for the use of our data!

15
What problems are we trying to solve?The view
from outside - MY
Audience
Summative external Accountability
Formative for improvement
Role
Research forimprovement
Educationalresearch
Academic, Professional
16
Academic/Formative known knowns and unknowns
IR as Research for Improvement
Academic formative and internal
Analysing (afresh?) existing data, or
investigating institutionally-focused research
questions, with a view to influencing practice.

Problem such activity is often scattered within
an institution, and unco-ordinated.
17
What problems are we trying to solve?
Research for improvement
Examples Student feedback, including
NSS Outcomes of quality assurance Assessment
results, cross-tabulated with demographics,
subject disciplines, etc. Incidence of
plagiarism Research assessment exercise
18
Academic/Summative known unknowns
IR as Educational Research
Academic summative and external, though with
some formative and internal relevance
Investigation of issues which are of wider
relevance than to a single institution, but which
may be triggered by an internal institutional
need.
Evidence needs to be particularly robust.
Potential for dissemination of findings beyond
the individual institution, in which case
criteria for peer review may well apply.
19
What problems are we trying to solve?
Educational Research
Examples Pedagogical innovations of various
kinds Assessment of achievement in workplace
settings Variation in honours degree
classifications across subject disciplines and
institutional types Relation of PT employment to
achievement on FT programmes Students
experiences of HE (FYE PT HE in FE,
etc) Non-completion
20
In the end what answer can we give to Gnothi
Sauton Know thyself?