How are research proposals to SSHRC evaluated? A look inside the black box of peer-adjudicated social science

1
How are research proposals to SSHRC evaluated?A
look inside the black box of peer-adjudicated
social science

• Charles Davis
• RTA/FCAD
• Ryerson University
• 15 September 2005
• Member, SSHRC committee 21, 2001-2004
• Chair, 2002-2004

2
Basic program features

• A Standard Research Grant (SRG) is intended to
  fund a 3-year research program
• Up to 250K over 3 years to individual or team
• Maximum 100k/year
• 2447 SRG proposal adjudicated
• 40.1 funded
• 28.9 of requested funds approved ( 80M)
• The success rate of new scholars is about 10
  lower than that of established scholars

3
21 adjudication committees (2004-5)

• Classics, ancient and mediaeval studies,
  religious studies, classical archaeology01
• History history of science, technology and
  medicine02
• Fine arts history and philosophy of art,
  architecture, theatre, music, film, dance03
• Linguistics, applied linguistics and
  translation05
• Economics07
• Sociology and demography08
• Geography, urban planning and environmental
  studies09
• Psychology10
• Education 1 Arts education, bilingual education,
  civic education, computer assisted instruction,
  counselling and career guidance, early childhood,
  educational psychology, environmental education,
  geography, health sciences education, history,
  mathematics, moral, values and religious
  education, pedagogy, physical education, reading
  and writing, science, second language, special
  education and vocational education (For
  additional disciplines, see Committee 17) 12
• Interdisciplinary and multidisciplinary studies15
• Anthropology and archaeology16
• Education 2 library and information science and
  archival science adult, continuing and community
  education comparative education, curriculum,
  distance education educational administration,
  planning, and governance history, philosophy
  theory of education higher education,
  measurement and evaluation, sociology of
  education, teacher education (For additional
  disciplines, see Committee 12)17
• Literature 1 English (from the Mediaeval to the
  Victorian period), French German Slavic18
• Literature 2 American, modern and contemporary
  literatures in English, English Canadian, First
  Nations, French Canadian Québec, romance, other
  languages literatures19
• Health studies and social work 20
• Human resources management, information systems,
  international business, management marketing,
  organizational studies business policy,
  industrial relations21
• Accounting, finance, management science,
  productions and operations management22
• Law, socio-legal studies and criminology23
• Political science and public administration24

4
golden rules of peer-adjudicated grantmaking

• The process is objective. It does not matter
  whether you have friends or colleagues on
  selection committees.
• The process is not random. It is not a form of
  lottery.
• Winning proposals are not selected.
• weaker proposals are eliminated from the
  competition the winners are those that remain.

5
The rules of the game perpetuate the Matthew
Effect

• Unto he that hath shall be given.From he that
  hath not shall be taken away

i.e. the funding mechanism obeys a law of
accumulated advantage
6
Scoring formula

• Regular scholar
• Record of achievement 60
• Research Program 40
• New scholar
• Whichever is higher
• Record of achievement 60, research program 40
• Record of achievement 40, research program 60

7
Research achievement

• evaluation of the record of research achievement
  is based primarily on contributions to research
  the applicant has made within the last six years
• if the applicant's research career has been
  interrupted, research achievement is evaluated
  based on his or her most recent period of
  research activity.
• For regular scholars, applicant's five most
  significant contributions are taken into account
  in order to accurately situate the most recent
  six years in the context of the applicant's
  overall career.

Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
8
Research achievement
Research contributions include

• refereed publications, including books, chapters
  of books and articles
• book reviews by the applicant/co-applicant or
  published reviews of his/her work
• research reports, papers presented at scholarly
  meetings or conferences, and other forms of
  written scholarly expression or participation in
  public discourse and debate which constitute a
  contribution to research
• where appropriate, contributions to the training
  of future researchers, including the supervision
  of graduate theses and/or the involvement of
  students in research activities
• research results from previous research grants,
  other awards from SSHRC or other sources
• academic awards and distinctions-new scholars may
  include scholarships and fellowships

Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
9
Research achievement
Evaluation criteria

• quality and significance of published work
  (taking into consideration the quality of the
  chosen publication venues)
• originality of previous research and its impact
  on the discipline or field
• quantity of research activity relative to the
  stage of the applicant's career
• demonstrated importance of other scholarly
  activities and contributions
• recentness of output (taking into account the
  nature of the applicant's career pattern and
  previous non-research responsibilities)

Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
10
Research achievement
Evaluation criteria

• importance and relevance of dissemination of
  research results to non-academic audiences (as
  appropriate)
• significance of any previous research supported
  by SSHRC or any other agency
• where applicable, contribution to the training of
  future researchers. (The committee must make
  allowances for applicants who have not supervised
  graduate students simply because their university
  does not offer graduate programs.)
• efforts made, where appropriate, to develop
  research partnerships with civil society
  organizations and government departments.

Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
11
Record of research achievement
12
Research program one or more projects over 3
years

• explicit objectives, situated within the context
  of current scholarly literature
• relationship of the proposed research to the
  individual's ongoing research or to insights
  gained from earlier achievements-,
• importance, originality and anticipated
  contribution of the proposed research
• theoretical approach or framework
• research strategies or methodologies (detailed
  methodology not necessary)

Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
13
Research program

• plans for the communication of research results
  within and beyond the academic community
• specific roles and responsibilities of students
  and research assistants, including how their
  duties will complement their academic training
• relationship of requested budget to proposed
  program of research.

Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
14
Research program evaluation criteria

• degree of originality and nature of expected
  contribution to the advancement of knowledge
• scholarly and intellectual as well as social and
  cultural significance of the research
• appropriateness of the theoretical approach or
  framework
• appropriateness and expected effectiveness of the
  research strategies or methodologies
• suitability and expected effectiveness of plans
  to communicate research results both within and,
  as appropriate, beyond the academic community

Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
15
Program of research
16
Score needed for funding
cutoff zone currently about 7.3 for SRGs
Meritorious but not funded
funded
rejected
Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
17
Achievement vs. research program
funded
Meritorious but not funded
rejected
Source SSHRC SRG Program Manual for
Adjudication Committee Members, Dec. 2001
18
Some common errors

• Theoretical framework weak or insufficiently
  explained
• Methodology weak or insufficiently explained
• A project is extended over 3 years to make it
  look like a program
• Budget is padded or poorly formulated
• Padding of CV
• me too proposals
• SSHRC funded research like this last year
• Another research project in already worked-over
  area
• Implausible teams

19
Some common errors

• Failure to respect page limits (6 pages means
  5.75-6 pages 6.1 pages is no good)
• Include literature review or information
  compilation as research
• Grad students roles not consistent with research
  program flow and objectives
• In a resubmission, failure to take into account
  the views of the committee and the external
  assessors
• Ultra cartesian or ultra baconian research designs

20
Risky storylines

• Im Too Important to Submit a Fully Worked-out
  Research Proposal my record speaks for itself
• Variant Were a team of Very High Profile
  Researchers. Our collective Research Achievement
  is off your scale
• The fate of the world hangs on the outcome of my
  project

21
Risky storylines

• My colleague got a grant last year to work on
  hamsters, so I will work on hamsters also
• It would please God if this proposal were
  funded
• My research results will overturn all
  established theories
• The Minister mentioned that this would make a
  great research project
• Because of the proliferation of incommensurable
  discourses in late postmodernity, you cannot
  understand what I am saying and I cannot
  understand my respondents, but I will study them
  anyway if you pay for it

22
A typical 3-year program

• Year 0 literature review completed methods and
  instruments selected preliminary hypotheses
  formulated
• Year 1 refine instruments and hypotheses through
  qualitative research (focus groups, grounded
  theory, etc.). Test instruments
• Year 2 apply instruments for data gathering
• Year 3 analysis, interpretation, modeling,
  dissemination of results

23
Common winning formulas for new scholars

• New scholar with good track record extends
  doctoral research via 3-year program
• Watch out. If the proposed research is too close
  to the doctoral research, it will be regarded as
  derivative. If it is too far away, it will be
  regarded as too bold.
• The most successful ones have published several
  articles (often with their PhD supervisor) before
  applying for a first grant

24
Common winning formulas for new scholars

• New scholar as PI with established scholar as
  co-investigator with specified roles
• The co-investigator brings up the research
  achievement score in proportion to his/her role
  in the project

25
Budgeting tricks and traps

• the committee may reduce your requested budget.
• It is good to ask for money for grad student
  stipends build grad students into your program
• Note It is best to use doctoral students. In
  regional universities it is OK to use masters
  students. If you use undergrads, make sure you
  have a good reason.

26
Budgeting tricks and traps

• Do NOT request conference travel money in Year 1.
• Hint OK to request modest funds for grad student
  travel to conferences, if they present.
• Do NOT inflate travel cost estimates.
• it is permissible to include travel costs of work
  with research collaborators, but not
  collaborators research costs
• Note that research travel costs include
  dissemination costs, which are also calculated
  separately

27
Budgeting tricks and traps

• Do NOT request funds for computers unless
  computers are clearly necessary for the research
  and they are unavailable through the university
• OK to ask for laptops for field research
• NEVER ask for funding for less than three years

28
Budgeting tricks and traps

• Research Time Stipends are only available if the
  home university contributes one-to-one matched
  funding
• Do NOT request funds for books. SSHRC does not
  like to pay for books.
• Be CAREFUL if you request funds for consultancies
  this is thin ice
• NOTE THAT once the budget is approved, SSHRC says
  that you can spend your grant however you like
  but your University controllers do not
  necessarily know this.
• At any rate, you cannot pay yourself an
  honorarium.

29
Budgeting tricks and traps

• See SSHRCs list of ineligible items. It
  includes training, purchase or rental of
  standard office equipment, preparation of
  teaching materials, entertainment and hospitality
  costs, research leading to a degree, fees and
  honoraria to colleagues, indirect or overhead
  costs, etc.
• ALWAYS include a clear explanatory budgetary note

30
Budget for hypothetical three-year, one-person
small project at a small or medium
(non-doctoral) university
31
Thank you!

• Questions or comments?