Research and Development Evaluation in Japan - PowerPoint PPT Presentation


PPT – Research and Development Evaluation in Japan PowerPoint presentation | free to download - id: 72b587-Y2VhN


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Research and Development Evaluation in Japan


Research and Development Evaluation in Japan & MEXT Naoko Okamura Director, Office of R&D Evaluation Science and Technology Policy Bureau Ministry of Education ... – PowerPoint PPT presentation

Number of Views:78
Avg rating:3.0/5.0
Slides: 39
Provided by: ysu74


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Research and Development Evaluation in Japan

Research and Development Evaluation in Japan
Naoko Okamura Director, Office of RD
Evaluation Science and Technology Policy
Bureau Ministry of Education, Sports, Culture,
Science and Technology (MEXT), Japan
?.Overview 1. Administrative Structure for ST
in Japan 2. National ST Strategy 3. Structure
of RD Evaluation in Japan 4. History of RD
Evaluation in Japan ?.RD Evaluation in MEXT 5.
Summary of the MEXT Guideline 6. Current System
of RD Evaluation 7. Evaluation System in
Competitive funds (Example with Special
Coordination Funds for Promoting RD 8.
Evaluation System of RIKEN 9. Evaluation System
  • ?. Overview

1. Administrative Structure for ST in Japan
(Since Jan. 2001)
Prime Minister
Cabinet Office
Minister of State (ST Policy)
CSTP ?Formulation of comprehensive
strategy ?Policy for allocation of resources such
as budget and human resources etc.
AEC NSC ?Designing, Planning, and coordinating
Presenting a Basic Policy, Comprehensive
?Formulation promotion of concrete RD
plans ?Coordination among relevant ministries -
Special Coordination Funds for Promoting ST -
Coordination on ST international cooperation
Public Corporation, Independent Administrative
Institution, National Research Institute
2. National ST Strategy
2-1 ST Basic Plan for 2001-2005
  • Three Basic Characteristics
  • - Creating and utilizing scientific
    knowledge to contribute to the world
  • - Developing international competitiveness
    and ability of
  • sustainable development
  • - Securing safety and quality of life
  • Basic Policies
  • - Strategic Priority Setting
  • - ST System Reforms
  • Doubling competitive research funds, introduce
    indirect costs to enhance competitive atmosphere
  • Improving mobility of human resources in ST
  • Increasing young researchers funds to exhibit
    their talent
  • Enforcing effective evaluation
  • - Internationalization

3. Structure of RD Evaluation in Japan
National Level
ST Basic Plan
General Guideline for RD Evaluation

Decided by the Prime Minister
Cabinet decision
Ministry Level
METI Guideline
MEXT Guideline
Other Ministries
Organization Level
RIKEN Guideline
JST Guideline
NEDO Guideline
NIAIT Guideline
Program for Major Activities
Program for Major Activities
  • CSTP is promoting RD evaluation as an integral
    part of ST policy
  • Ministries and RD Organizations carry out the
    major part ot evaluation activities

  • 4. History of RD Evaluation in Japan
  • 1997 First National Guideline
  • - Introduced RD evaluation
  • - Encouraged external
  • - Aimed at efficient resource
  • 2001 Revised National Guideline
  • - To upgrade fairness and
  • - To strengthen link with
  • - To establish resource base
    for evaluation
  • 2005 Revised National Guideline
  • - To encourage researchers
    challenge for creativity
  • - To develop evaluation skills
    to increase reliability
  • - Evaluation results should be
    utilized more strictly
  • MEXT has just established Guideline for
    Evaluation of Research and Development in MEXT
    in Sep.2005

  • ?. RD Evaluation in MEXT

5. Summary of the MEXT Guideline
5-1 Role
- Sets out the basic approach for conducting
evaluations on RD which fall under MEXTs
jurisdiction. Internal divisions of MEXT should
conduct evaluations based on this guideline while
RD institutions should refer to this guideline
for conducting appropriate evaluations
5-2 Basic Concept
  • Goals of the evaluation
  • To promote
  • Research developing new frontier of ST
  • Research contributing to the society/economy
  • Significance of the evaluation
  • Realizing open, flexible and competitive RD
  • Allocating RD resources efficiently and
  • Obtaining public support through disclosure of
    RD achievements

  • 5-3 Direction of Reform
  • Encouraging researchers challenge for
    creativity, promoting new frontier research
  • Recognize that former evaluation sometimes
    discourage researchers
  • Continuation of the excellent projects through
  • Announce the results of reflection
  • Ensuring the evaluation resources and improving
    evaluation system
  • Prepare necessary budget for evaluation process
  • Foster personnel for evaluation by training
  • Develop the National RD database
  • Develop efficient evaluation
  • Eliminate too much procedure and too many data
    for evaluation

  • 5-4 Common Principles
  • Setting the specific objective and aim of the
  • Selection and assignment of reviewers
  • Avoid conflict of interest
  • Timing of evaluation
  • Follow-up activity after the evaluation
  • Method of evaluation
  • Announcing the procedure, criteria, etc. to the
  • Evaluating quality rather than quantity
  • Expanding objective indicators use taking
    consideration for their limit
  • Revealing criteria for evaluation
  • Securing flexibility and avoiding burden
  • Utilization of the results
  • Reflection to the distribution of resources

  • 5-5 Category of Evaluation
  • Research and Development Policy
  • Program, system etc.
  • Research and Development Theme
  • Competitive fund
  • Project
  • Research and Development Institute
  • Independent Administrative Institutions
  • National Laboratories
  • Public Corporations
  • Universities
  • Achievement of researcher

6. Current System of RD Evaluation (1)
CSTCouncil for Science and Technology SACSpace
Activity Commission

6. Current System of RD Evaluation (2)
IAIIndependent Administrative Institute NUCNatio
nal University Corporation IAIEC Independent
Administrative Institute Evaluation
Committee NUCEC National University Corporation
Evaluation Committee MPHPTMinistry of Public
Management ,Home Affairs,Posts and
7. Evaluation System in Competitive funds -
Special Coordination Funds for Promoting RD -
POINTS ?SCF is fund for developing new frontier
of ST and contributing to the society/economy.
? Evaluation for various aspects . Scientific
evaluation by Peer reviews Final Selection
by Expert Panel (including industry
researchers) ? Working Groups are supported by
Scientific stuff Program Officers,Program Director
Ex-post Evaluation
Screening (Prior Evaluation)
examination (consultation)
Selection of subjects
3rd year
Acceptance of applications
Documentary examination
RD ( 5 years )

by CST
WGs, are established in the every fields
and subjects are examined.
Interim Evaluation
Peer reviews
Working groups (expert panel)
established in JST(Japan ST corporation)
8. Evaluation System of RIKEN 8-1 Profile of
RIKEN Law (Independent Administrative
Institution founded under Law No. 160, December
13, 2002)
Basis for Establishment
October 1, 2003
Date of Establishment
To conduct comprehensive research in science and
technology (excluding humanities and social
sciences) and to foster advances in scientific
research and technological standards
83,956 million (of which 74,920 million is
funded from the government subsidy) 86,796
million (of which 75,882 million is funded from
the government subsidy)
2004 budget 2005 budget
2,835 full-time employees (of which 692 are
executives and Mandatory Retirement Age Contract
Employees as of end of 2004
Ryoji Noyori, Dr. Eng.
  • (1) Comprehensive research and testing
  • basic science research (b) promotion of
    accelerator scientific research (c) promotion of
    bioresources businesses (d) promotion of
    synchrotron radiation research (e) promotion of
    integrated collaborative in research (f)
    promotion of comprehensive brain science
    research (g) comprehensive genome science
    research (h) promotion of botanical science
    research (i)comprehensive research in
    developmental biology (j) promotion of SNP
    research (k) promotion of comprehensive research
    into immunology and allergy science and (l)
    promotion of strategic research
  • (2) Dissemination and active utilization of
    research achievements
  • Presentation of research reports, access to
    resources in biogenetics, establishment of
    intellectual property rights on scientific
    achievements, licensing of patents, etc.
  • (3) Shared-use of facilities and equipment
  • (4) Training of researchers and engineers and
    advancement of their qualities and skills
  • (5) Promotion of shared use of Spring-8

Principal operations
8-2 Eight Features of RIKEN Operations
1. Longstanding reputation as a comprehensive
research institute
5. Competitive and diversified research
History of 88 years since 1917 Japans only
comprehensive research institute of natural
Acquisition of external funding that is
competitive and diverse Creation of
competitive environment for funding inside the
2. Broad range of research
6. Open research system
Large number of fields (physics, chemistry,
engineering, biology, medicine,
etc.) Interdisciplinary research From basic
science to applied research
Joint research and broad-ranging personnel
exchanges with universities, business
enterprises, etc. Cooperation Program for
graduate school with universities in Japan
7. International Institute
3. Versatile research organization adapted to
the nature of research
Large number of foreign researchers Three
overseas research centers Cooperation with
foreign research institutes (including NIH
Max Planck)
Institution Laboratory (for emerging research
area, compound research and integrated research
areas hub organization) Frontier Research
System (new areas of research
target-oriented) Center system (centralized
action in prioritized fields) Availability of
research platform (bioresources, SPring-8, etc.)
8. Research evaluation on multiple stages
4. Promotion of alliances with industry and
technology transfer
First systematic evaluation by external sources,
including foreign researchers Evaluation on
research unit revels (by Centers, by
laboratories, etc.) Evaluation of research
themes research performance
Joint research, patent licensing, etc.
8-3 RIKEN Organization Research Centers
8-4 Evaluation System of RIKEN
RIKEN Overview
RIKEN Advisory Council (RAC) Evaluation of RD
RAC advisory body to the President on overall
research activities and management in RIKEN
Discovery Research Institute Harima Institute
Evaluation of RD Organizations
Recommendation to Center Directors on research
activities and management of Center.
(Advisory Council)
Internal and external evaluation by research
themes (internal evaluation since 1965)
Evaluation by external assessor of each research
group as a unit
Evaluation of RD Themes
External evaluation of research laboratories
(research performance review since 1972)
Evaluation conducted by Head of each research
unit at the end of each fiscal year, when
contracts are renewed.
Evaluation of Researchers performance
Internal evaluation committee to assess promotion
for researchers (ex. research ? senior researcher)
The Evaluation System is essential to make
appropriate judgments on RD activities, hence to
increase work efficiency and vitalize research
activities to achieve higher goals.
8-5 Advisory Council System of RIKEN
RIKEN Discovery Research Institute Harima
Institute Directors, DRI HI
RIKEN Advisory Council
RIKEN Frontier Research System Director,FRS
RIKEN Brain Science Institute Director,BSI
RIKEN BioResource Center Director,BRC
RIKEN Genomic Sciences Center Director,GSC
RIKEN Plant Science Center Director,PSC
RIKEN SNP Research Center Director,SNP
RIKEN Research Center for Allergy and
Immunology Director,RCAI
President The Board of Executive Directors
RIKEN Center for Developmental Biology Director,CD
8-6 Distinctive Features of the Advisory Council
  • Advisory Council (AC) Members are carefully, as
    well as internationally, selected from different
    academic fields to cover RIKENs broadly
    conducted research activities.
  • ? More than half of the members are foreign
    nationals. All five RAC meetings of the past
    were chaired by foreign nationals.
  • AC chairman of each Center to participate in the
    RAC meetings as a member.
  • ? Appropriate for RAC which conducts umbrella
    evaluation of RIKEN.
  • Reports are produced only by AC members so that
    activities in Centers are directly and openly
  • ? Assurance of objectivity and fairness
    (however, RIKEN might present its arguments
    vis-à-vis those reports)
  • Responses to the RAC recommendations are compiled
    in a report immediately after the RAC meeting,
    which is sent to RAC Members. The final
    progress of actions taken to respond to the RAC
    recommendations is to be evaluated at the next
    RAC meeting.
  • ? Recommendation ? Response ? Reporting
    makes a perfect flow to contribute to the
    RIKENs development.

8-7 Advisory Council Status (Past 5 Years)
RAC (RIKEN Advisory Council) Fourth meeting 4-7 June 2000 Fifth meeting 6-9 June 2004
ILAC (Institute Laboratories AC) First meeting 6-9 February 2000 Second meeting 1-4 February 2004
FRAC (Frontier Research System AC ) First meeting 11-12 May 2000 Second meeting 7-8 June 2001 Third meeting 6-8 April 2004
BSAC (Brain Science Institute AC) Third meeting 12-14 April 2000 Fourth meeting 4-6 April 2001 Fifth meeting 18-20 September 2002 Sixth meeting 12-14 April 2004
BRAC (BioResource Center AC) First meeting 1-3 March 2004
GSAC (Genomic Sciences Center AC) First meeting 21-23 March 2000 Second meeting 30 November - 2 December 2003
PSAC (Plant Science Center AC) First meeting 12-14 March 2003 Second meeting February - March 2004 (mail review)
SRAC (SNP Research Center AC) First meeting 30 May 2001 Second meeting 11-13 December 2002 Third meeting 8-10 February 2004
AIAC (Research Center for Allergy and Immunology AC) First meeting 26-28 April 2004
DBAC (Center for Developmental Biology AC) First meeting March 2001 (mail review) Second meeting 23-25 April 2002 Third meeting 1-4 June 2004
8-8 What is the RAC?
  • The RIKEN Advisory Council conducts an overall
    review of RIKEN and makes recommendations and
    proposals to the President on managerial issues
    including management of the Board of Executive
  • Recommendations on policies to further vitalize
    research activities
  • Identifying research areas where resources should
    be increased and where further development is
  • Recommendations on new areas of development,
    especially on how to embark on unexplored fields
  • Suggesting a policy to promote internationalizatio
    n in RIKEN and international cooperation by RIKEN
  • Any other recommendations on other issues and
    measures for the advancement of RIKEN

Meetings First meeting 21-24 June 1993 Second
meeting 26-29 June 1995 Third meeting 31 May
- 5 June 1998 Fourth meeting 4-7 June
2000 Fifth meeting 6-9 June 2004 Sixth
meeting 6-9 June 2006 (tentative)
8-9 Terms of Reference to the 5th RAC
Meeting - Noyori Initiative -
  • Visibility of RIKEN
  • Improve the image and public recognition of RIKEN
  • Researchers and administrative staff should make
    the public aware of the importance of science and
  • 2. Maintaining RIKENs Outstanding History of
    Achievement in Science and Technology
  • Sustain and build the RIKEN research spirit
  • Emphasize quality of research. Maintain
    excellence and prestige of the RIKEN brand
  • Strengthen action further to achieve intellectual
    properties and to make contributions to society
    and industry
  • 3. RIKEN that Motivates Researchers
  • Curiosity-driven concepts
  • Presenting challenges that are unique and high in
    risk level
  • Fostering promising human talent
  • 4. RIKEN that is Useful to the World
  • Close ties with industry and society
  • Science and technology that supports civilized
    society (beyond the boundaries of universities or
  • 5. RIKEN that Contributes to Culture
  • RIKEN must raise the level of its own culture
  • Dissemination of knowledge to humanities and
    social science

8-10 RAC Recommendations of the 5th RAC Meeting
1. Build a strong scientific vision for the
future of RIKEN 1a) Revise the scientific
governance structure 1b) Develop a long-term
plan for the support of basic research activities
at RIKEN 2. Strengthen the role of RIKEN
President 2a) Set up an external advisory board
reporting to the president 2b) Strengthen the
position of the center and institute directors 3.
Increase efforts to build strategic
relationships 3a) Broaden the scope of internal
strategic program 4. Develop programs to increase
the quality of post-doctoral staff and graduate
students working at RIKEN 4a) Create a
pre-doctoral fellowship program to support
graduate students working at RIKEN 4b) Provide
support for all staffers seeking employment after
their RIKEN contracts ends 5. Reassess RIKENs
technology transfer regime 5a) Develop a
strategic framework to support translational
research 6. Increase the number of foreign
scientists working at RIKEN 6a) Increase the
number of Japanese women scientists in leading
positions at RIKEN 7. Implement best practices
in management and administrative services 7a)
Develop a coherent long-term personnel management
strategy 7b) Develop best practices for the
management of large facilities and resource
8-11 Issues under Evaluation
Pressure from Evaluation Evaluation imposes
pressure on researchers, especially in research
organizations consisting of Fixed-term Contract
Researchers. This seems to be increasing tensions
among researchers beyond what is necessary. ?
Discourage researchers to take on challenges
Combine information overlapped among various
evaluation activities The common information such
as research performance is repeatedly gathered by
different persons for various evaluation
activities as the information is required at a
different timing and by different evaluation
organizations, i.e. IAI (Independent
Administrative Institution) Evaluation and RAC
Meeting. Having that in mind, it is advisable
to establish an efficient data-management system
for combining and/or sharing such common
information to avoid duplicate work and reduce
workload of both researchers and back-office
9. Evaluation System of NIMS 9-1 Profile of
NIMS Law (Independent Administrative Institution)
Basis for Establishment
April 1, 2001
Date of Establishment
1. Conduct Basic and Generic and
Infrastructural research 2. Promote
dissemination of research outcomes and their
utilization 3. Open advanced facilities and
equipments to outside researchers 4. Educate
materials scientists and oegineers
173.5 million (of which 146.5 million is funded
from the government subsidy and 23.2 million is
income for entrusted research (ex. Competitive
2005 budget
1,538 full-time employees (of which 692 are
permanent staff)
Employees as of end of 2004
KISHI Teruo, Dr. Eng.
Nanomaterials Safe Materials Environment and
Energy Materials Improvement of Research and
Intellectual Infrastructure
Focused RD areas under Five-Year Program at
MINS (2001-2006)
9-2 Organization of NIMS
Advanced Materials Laboratory
Nano-materials Laboratory
Materials Engineering Laboratory
Biomaterials Center
Superconducting Materials Center
Computational Materials Science Center
Nanotechnology Researchers Network Center of Japan
Steel Research Center
Ecomaterials Center
International Center for Young Scientists
High Magnetic Field Center
Materials Information Technology Station
Material Analysis Station
Cooperative Graduate School NIMS-Course, Univ. of
High Voltage Electron Microscopy Station
9-3 Schematic Image of Evaluation System in NIMS
3. MIAC Evaluation Committee for Policy and IAI
1. Evaluation of NIMS
by MEXT IAI Committee (Annual Evaluation of the
Mid-term Program Achievement)
Secretariat MEXT
4. R D Evaluation
Submit self-rating sheets
based on R D Evaluation Procedure
2. Making Self-rating Sheets by NIMS
Secretariat ISO and REO
i) RD Projects by External Funds
by Funding Agency
Evaluation Results are Reflected on Management
and R D Activities through Reallocation of
Budget by the Executive Meeting
ii) Major Research Projects
by External Committee (advance, mid-term, ex
post facto and follow-up)
iii) Exploratory Researches
by Director General of Each Unit (advance and ex
post facto)
5. Researchers Personal Evaluation
6. NIMS and Research Unit Evaluation
Secretariat REO
based on Researchers Personal Evaluation
by the NIMS Advisory Board
iv) RD projects by NIMS Competitive Funds
Secretariat ISO and REO
Secretariat ISO
by Internal Committee (advance and ex post
Use of Evaluation Results
Secretariat ISO
Reflection of Evaluation Results
MIAC Ministry of Internal Affairs and
R D Research and Development
MEXT Ministry of Education, Culture, Sports,
Science and Technology
ISO Integrated Strategy Office
IAI Independent Administrative Institution
REO Research Evaluation Office
9-4 Headings of NIMS evaluation as an
independent administrative institute (1)
  • Management of research organization
  • Improving the service to the people and other
  • I. 1. Basic research and development
  • 1. 1 Major research projects
  • I. 1. 1. 1 Nanomaterials
  • I. 1. 1. 2 Environment and energy materials
  • I. 1. 1. 3 Safe materials
  • I. 1. 2 Improvement of research and
    intellectual infrastructure
  • I. 1. 2 1) Improvement of research
  • I. 1. 2 2) Improvement of intellectual
  • I. 1. 3 Exploratory researches
  • I. 1. 4 Proposal to external funds and
    acceptance of entrusted researches

I. 2. Dissemination of research outcomes and
their utilization
I. 2. 1 Dissemination of research outcomes and
promotion of public relations
I. 2. 1 ? Research products
I. 2. 1 ? Public relations
I. 2. 1 ? Publishing materials data bases and
data sheets
I. 2. 2 Promotion of technology transfer
Number of patent applications Number of licensed
I. 3 Making facilities and equipments open to
non-NIMS researchers
I. 4 Training and education of researchers and
I. 4. 1 Reception of trainees
I. 4. 2 Participation and dispatch of lecturers
to scientific societies and academic meetings
Number of participants to the scientific
societies and academic meetings Number of
lecturers dispatched to scientific societies and
academic meetings
I. 5 Others
I. 5. 1 Improvement of survey and coordinate
I. 5. 2 Research exchanges
I. 5. 2 1) Implementation of joint research and
promotion of cooperation
I. 5. 2 2) Reception of researchers from other
I. 5. 2 3) Dispatch of NIMS researchers to other
I. 5. 3 Cooperation in accident-cause-finding
II. Steps to achieve goals for efficient
II. 1. Organizational and management systems
II. 1. 1 Fundamental policy for research
II. 1. 1 ? Construction of research system
II. 1. 1 ? Construction of research
support system II. 1. 1 ? Construction of
technology transfer system
II. 1. 2 Fundamental policy for management
II. 1. 2 1) Expansion of project leaders
II. 1. 2 2) Rational arrangement of human
II. 1. 2 3) Efficiency of management
II. 1. 2 3) ? Simplicity, rapidity and efficiency
of administrative procedures

II. 1. 2 3) ? Outsourcing

II. 1. 2 3) ?
Efficient management of operating subsidy
III. Budget, income and expenditure and fund plan
IV. Amount of short-term loan payable
V. Plan for disposal or mortgage of important
assets if planned
VI. Use of surplus fund
VII. Others concerning management specified in
MEXT ordinance
VII. 1. Planning for facilities and equipments
VII. 2. Planning for personnel affairs
9-5 Outline of Evaluation of Major Research
(The NIMS rule is based on the government outline
and MEXT guidelines for evaluation.)
Times of evaluation Advanced/ mid-term (for
5-years projects)/ ex post facto/ follow-up (for
selected projects)
(25 Major research projects are evaluated.)
Members of evaluation committee
(1) Candidates of
about 10 specialists outside NIMS are proposed.
(2) Persons concerned are excluded.
(3) 4 or 5 are to
be selected.
Preparation Materials for evaluation and
evaluation sheets are sent beforehand to
committee members.
Evaluation committee Project leaders explain the
aim, objective, plan, results and so on.
Committee members and the project leaders have
Evaluation results Evaluation sheets compiled by
the chairperson are submitted to the project
leaders. If the leader is against the results,
discussion will be held between committee members
and the project leaders through the secretariat.
Reflection of results The president of NIMS
confirms the final evaluation results. The
results are reflected on the budget and to
improve the project.
Official announcement The final evaluation
results are disclosed to public via internet.
9-6 Performance-based Personnel System
  • Objective
  • Maximize the research achievements under a
    competitive environment to accomplish the
    institutes goals
  • Framework
  • Develop achievement-oriented pay and
    results-driven achievement systems
  • Reward high-performing researchers
  • Achievements to evaluate
  • 3P (Papers, Patents, Products)
  • Scored automatically by on-line system
  • Paper 3 Impact factor x 2
  • Activities contributed to NIMS missions except
    3P (management, academic activities,
    collaborations, research assistance, etc.)
  • Rated by supervisor

9-7 Performance-based Pay System
  • Both monthly pay and bonus reflect Individual
  • Performance-based pay determined by individual
    achievements over last several years
  • Director-general 18 40 of basic pay
  • Director 14 25
  • Senior researcher 12 20
  • Performance-based bonus determined by individual
    achievement of the year before
  • Director 4.25 months a (average 0.4 months)
  • a is proportional to individual achievement point