Models%20of%20Evaluation%20For%20Research%20Proposals%20in%20Turkey

1
Models of Evaluation For Research Proposals in
Turkey

• Prof. Dr. Omer CEBECI
• Vice President Funding, TUBITAK
• (The Scientific and Technical Research Council of
  Turkey)
• S. GENC, A. KERC, H. KARATAS, A. FEYZIOGLU,
• F. COSKUN, O. OZPEYNIRCI, B. DIKMEN,
• G. KOZANOGLU, H. GULER O. KARA

2

• SUMMARY
• Research funding in Turkey, TUBITAK recent
  boost
• Three-dimensional evalution used by TUBITAK for
  the evaluation/selection of research project
  proposals grouped under three categories
• (1) curiosity driven academic research, and
• (2) customer driven applied research,
• both in universities and research institutions,
  and
• (3) technological and innovation driven research
  conducted by the private industry, plus
• (4) research equipment infrastructure.
• Details of the Phrase-anchored rating scale

3
Actors of Turkish National Science and Technology
System (policy makers)
TUBITAK-The Scientific and Technical Research
Council of Turkey
BTYK- Supreme Council of Science and Technology
MIT-Ministry of Industry and Trade
TAEK- The Turkish Atomic Energy Commission
TOBB-Union of Chambers and Commodity Exchange of
Turkey
ME-Ministry of Education
HEC-Higher Education Council (YÖK)
The Turkish Academy of Sciences (TÜBA)
TTGV- Technology Development Foundation of
Turkey
KOSGEB- Small and Medium Sized Industry and
Development Organizations
TÜRKAK- Accreditation Board
DPT- State Planning Organization
DTM-Foreign Trade and Treasury
4
Actors of Turkish National Science and Technology
System (science and research performers)

• TUBITAK National Research
• Centers and Institutes
• Marmara Research Center (MAM)
• Institute of Energy
• Chemistry and Environment Institute
• Food Institute
• Materials Institute
• Information Technologies Research Institute
• Earth and Marine Sciences Research Institute
• Information Technologies and Electronics
• Research Institute (BILTEN)
• National Electronics and Criptology
• Research Institute (UEKAE)
• Defence Industry Research and Development
• Institute(SAGE)
• Basic Science Research Institute
• National Academic Network Center
• Genetic Engineering
• and Biotechnology Research Institute

Other National Research Centers and Institutes
Turkish Industry
Turkish State Universities
Turkish Private Universities and Research
Centers
5
Turkish National Science, Technology and
Innovation System
President of Republic
Prime Minister
BTYK
STB
MEB
YÖK
Universities
TÜBITAK
DPT
TÜBA
DTM
DIE
KOSGEB
TPE
TTGV
TOBB
TSE
TÜRKAK
RD Institutions
BTYK- Supreme Council of Science and
Technology STB- Minister of Industry and
Trade MEB- Ministry of National
Education TÜBITAK- The Scientific and
Technological Research Council of Turkey YÖK-
Higher Education Council (YÖK) TÜBA- The Turkish
Academy of Sciences (TÜBA) DPT- State Planning
Organization TÜRKAK- Accreditation Board
TPE- Turkish Patent Institute TSE- Turkish
Standards Institution DIE- State Institute of
Statistics DTM- Undersecretariat of the Prime
Minister for Foreign Trade TOBB-Union of
Chambers and Commodity Exchange of
Turkey KOSGEB- Small and Medium Industry
Development Organizations TTGV- Technology
Development Foundation of Turkey
6
National ST Indicators
Number of triadic patents per million population
2,5
2
Total RD personnel per 1000 employment
1,5
Number of papers per million population
1
0,5
0
GERD as a percentage of GDP
GERD per capita population
7
National ST Indicators
Number of triadic patents per million population
1
Number of papers per million population
0,5
Total RD personnel per 1000 employment
GERD as a percentage of GDP
GERD per capita population
8
Scientific Publications
9
Knowledge as a Tool

• Knowledge is a tool that can be used for a
  variety of social objectives,
• including
• Meeting Basic Human Needs
• Increasing Safety Security
• Improving the Quality of Life
• Economic Growth and Development

10
Knowledge - RD Cost per Kg
11
Share of TUBITAK inTR Fiscal Budget

12
Research Proposals

• from universities and research institutions
• curiosity driven academic research
• customer driven applied research
• from private industry
• technological innovation driven research

13
TUBITAK Research Project Proposals
Science and Technology Project Proposals
14
Running Research Projects
15
EVALUATION MEASUREMENT
16
TARIHTEN BUGÜNESAGLIK ve ÖLÇME

• Ates ve nabiz 1625
• (Santario ve Galileo)
• Torricelli - 1643
• Tansiyon - 1733 (Hales)
• 1896 - Riva-Rocci
• Korotkov - 1905
• Ates - nabiz - tansiyon ölçüm ve kayitlari ile
  hasta izleme - 1920

17
ÖLÇME ve DEGERLENDIRME

• Ölçemediginizi anliyamazsiniz !
• Lord Kelvin (1824 - 1907)
• (If you cant measure it -
• you dont understand it)
• Her hesaba katilmasi gerekenin
• sayilir olmasi gerekmez !
• Albert Einstein (1879 - 1955)
• (Not everything that can be counted counts not
  everything that counts can be counted)

18
Çikti mi ? Sonuç mu ? Etki mi?
Çiktilarin sonucu olarak gerçeklesen ETKI !
19
OUTPUT OUTCOME IMPACT IMPACTS
achieved as consequences of the OUTCOMES
resulting from the realisation of the OUTPUTS
20
OUTPUT OUTCOME IMPACT IMPACTS
achieved as consequences of the OUTCOMES
resulting from the realisation of the OUTPUTS
( My eye jelly project )
21
Intended / Expected / Desired IMPACT . . . in
full control of my mandate . . . Unintended /
Unexpected / Unwanted BUT ACTUAL REAL IMPACT
. . . under strict control of my wife ! . .
22
Evaluation Criteria

• three-dimensional
• evaluation criteria
• developed in collaboration with
• researchers reviewers

23
1) Curiosity DrivenAcademic Research- the three
dimensions -

1. intellectual/scientific/professional merit
2. expected impact of the anticipated outcome
3. achievability of the research with the proposed
   research team, equipment/facilities and methods

24
2) Customer Driven Applied Research- the three
dimensions -

1. research and development merit
2. expected impact of the anticipated outcome
3. achievability of the research with the proposed
   research team, equipment/facilities and methods

25
3) Technological InnovationDriven Research-
the three dimensions -

1. technological level of the research
2. innovative level of the product/outcome
3. feasibility of the process

26
Evaluation Process

• all three-criteria are given equal weights
• Phrase-Anchored Rating Scale
• versus
• Likert Scale
• sub-criteria phrases describing
• very competitive competitive
  not-competitive
• features of the proposal

27
Very competitive

• scientifically and professionally outstanding
  very well justified project
• points to an opportunity for a major
  contribution to the advancement of the knowledge
  and/or to the resolution of a problem of
  practical importance

28
Competitive

• scientifically and professionally competent and
  justified proposal which will make a contribution
  to the advancement of knowledge /or the
  resolution of a problem of practical value
• therefore
• support is suggested if funds are available

29
Not competitive

• work routine in character
• scientifically and professionally unsatisfactory
  and poorly organized

30
Evaluation Process
5-8 Individual Reviewers

evaluate 10-15 proposals by referring to the
sub-criteria phrases
Panels
a meeting for a final verdict
31
Curiosity / Merit V. Comp.

• Outcomes have high potentials for publication in
  journals or books listed in international indexes
• The originality of the work has been supported by
  extensive and critical literature survey
• Hypothesis for evaluating the research topic is
  very well defined
• Explanation and analysis of the expected outcomes
  reveal the superiority of the work in comparison
  to the existing science and technology.

32
Curiosity / Merit Not comp.

• Scientific consistency and the rationale of the
  research are not clearly explained.
• A clear scientific / technological question is
  not put forward.
• Project is more like an investigation / data
  collection / routine work.
• References provided include similar studies
  literature survey does not point out the basis /
  importance of the project.
• The research is based on unreliable data and
  hypothesis.

33
Curiosity / Impact V. Comp.

• Very important in terms of sustainable
  development of the country /very likely to be
  implemented to find solutions for the problems of
  society
• Very likely to be employed in different
  scientific technological fields
• Very likely to generate new projects
• Commercialization potential of the outcomes is
  very high
• The project is supported by international,
  national or industrial sources.

34
Curiosity / Impact Not comp.

• Potential for adding value to science and
  technology is low
• Subject of the project is not among the
  priorities of the country
• Not likely to result in intellectual property
  worth-protecting.

35
Curiosity / Achiev. V. Comp.

• Project Team
• The team is experienced in national /
  international projects related with same/similar
  subject
• They have experience as advisors /authors /
  referees / editors / book authors
• The can allocate enough time for the project
• Competencies / responsibilities / roles of the
  team members are well defined and adequate
• End users of the project outcomes are also
  members of the team.

36
Curiosity / Achiev. Not Comp.

• Project Team
• The team is not experienced in conducting
  projects of this size
• The knowledge and awareness of the team is not
  sufficient
• They do not have important publications in the
  subject of the project
• Some of the team members are irrelevant for the
  project
• Essential competencies are lacking

37
Curiosity / Achieve. V. Comp.

• Infrastructure
• Infrastructure of the institution is very
  adequate for the project
• Additional equipment requested within the scope
  of the project is very compatible with the
  existing infrastructure and the project
• Existing sources / equipment are used rather than
  purchasing new sources
• Requested equipment can also be used in other /
  future projects

38
Curiosity / Achieve. Not Comp.

• Infrastructure
• Infrastructure of the institution is not adequate
  for the project, unless supported with major
  equipment
• Equipment requested within the scope of the
  project is not compatible with the existing
  infrastructure and the project

39
Curiosity / Achieve. V. Comp.

• Methodology
• Approach / methodology are very well designed to
  reach the target
• Methodology is correct and well-defined and
  standard methods and literature are cited
• Preliminary experiments have been conducted to
  rationalize the hypothesis
• Alternatives (plan B) have been considered if
  difficulties are encountered

40
Curiosity / Achieve. Not Comp.

• Methodology
• Methodology is not adequate to reach the target
• Relations between the experiments and hypothesis
  are not well defined
• Methodology is not explained with a common
  scientific basis
• Possible problems and limitations are not
  considered
• Statistical analytical requirements are not
  considered

41
Curiosity / Achieve. V. Comp.

• Timeline
• Proposed period time schedule are realistic
• Budget
• Proposed budget is realistic and well-justified
• Project is also supported by other institutions

42
Curiosity / Achieve. Not Comp.

• Timeline
• Proposed period and time schedule are not
  synchronized
• Time schedule is not adequate
• Budget
• Budget is not well-define and requested amount is
  too low / high
• There is no possibility of support from other
  institutions

43
Customer / Merit V. Comp.

• Aims to develop a national / international novel
  technology (methodology, system, product, process
  / technique)
• Brings comparable superiority to the existing
  system
• Outcome will be a technology to be protected
  under the intellectual property rights
• Work consists of a scientific and rationale
  approach
• An interdisciplinary project with the
  collaboration of the Public- Private
  Industry-University-Research Institutions

44
Customer / Merit Not comp.

• Literature survey and market survey are not
  satisfactory, work is based on unreliable data
• No scientific / technological rationale and
  integrity in the project
• It is a study that had been done before in the
  country
• Aim, objective and motivation are not clear
• Outcome is not qualified as applicable/usable
• Not related with a real need / problem

45
Technological Level V. Comp.

• Technology / product developed aims to fill a gap
  in the existing technology or replace the
  existing technology within the following 3 years
• Has an interdisciplinary approach to solve more
  than one problem
• Rationale of the RD is well established
  (theoretical / analytical / experimental)
• Added value of the anticipated outcome of the RD
  project is considered
• Has contribution to increase RD staff
• A doctorate / masters study is incorporated
  within the project and the outcomes have
  potentials to be published in national/internation
  al journals.

46
Technological Level Not comp.

• Literature survey and market survey are not
  adequate
• No scientific/technological advance and integrity
  in the project
• RD rationale (analytical and /or experimental)
  is not adequate
• Aim, objective and motivation are not clear.-
  Procedure that has been used will not provide a
  change / improvement in the methodology and
  technology

47
Innovation Level V. comp.

• Outcomes may lead to spin-off company
• Outcomes include production standards and
  technical specifications
• Outcomes will affect other sectors as well
• Outcomes very likely to increase the export
  capacity of the country
• Very likely to be a basis for generating new
  projects / products

48
Innovation Level Not comp.

• Outcome is not qualified as applicable/usable
• Period of usefulness is very limited or not
  effective
• Does not provide new areas of work and as a
  result does not increase employment
• Expenditure for the RD study is far beyond the
  expected economical benefit of the product
• Support provided by the private organization is
  not sufficient.

49
Comments

• Reviewers (hundreds)
• The model reduced the,
• burden of the review process on them,
• subjectivity and variability of the opinions of
  the individuals
• Convenience of selecting from a comprehensive
  list of phrases as well as
• the freedom of offering their own judgments
• Researchers (thousands)
• Sub-criteria phrases to be very instrumental in
  guiding the development of their proposals

50
Thank you

• Hundreds of scientists, researchers and reviewers
  participated in workshops for the development of
  the criteria and phrases listed in the tables.
• Their contributions are gratefully acknowledged.