Nuove frontiere e prospettive di finanziamento per la ricerca europea nel periodo 20072013

1
Nuove frontiere e prospettive di finanziamento
per la ricerca europea nel periodo 2007-2013

• Valutazione delle proposte e probabilità di
  successo
• FP6 IST

Giancarlo Abbate
2
Sommario

• La valutazione delle proposte Area Tematica del 6
  PQ Tecnologie per una società dellinformazione
• Le procedure di valutazione
• I criteri ed i sottocriteri di valutazione per i
  differenti strumenti di finanziamento
• Il processo di selezione delle proposte
• Gli esperti indipendenti. Lettura individuale
• Consensus meeting
• Lettura incrociata. Panel meeting. Classifica.
  Audizione
• Commenti sul processo di selezione
• Paragone con le procedure di valutazione di
  alcuni altri programmi.

3
Le procedure di valutazione

• Procedura a due stadi esempio FET-Open STREP

ftp//ftp.cordis.lu/pub/ist/docs/fet/int-o-1.pdf
4
Le procedure di valutazione

• Procedura a due stadi esempio FET-Open STREP

5
Le procedure di valutazione

• Procedura a due stadi esempio FET-Open STREP

6
Le procedure di valutazione

• Procedura a due stadi esempio FET-Open STREP

7

• Procedura a singolo stadio Esempio
  FP6-IST-2002-2.3.2.2 Optical, opto-electronic,
  photonic functional components

http//www.europa.eu.int/comm/dgs/information_soci
ety/evaluation/pdf/5_y_a/ist_5ya_final_140105.pdf
The scientific and technical evaluation of
proposals is carried out by external experts from
the public and private sector. Full proposals are
evaluated, ranked and selected. The IST theme
under FP5 and FP6 generally has a high
oversubscription of the budget e.g. the first
call for the IST Priority Theme in FP6 attracted
1400 proposals requesting 6.2 B, whereas the
available budget was only 1.07 B.
8
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• Common evaluation criteria for evaluating
  proposals
• A number of evaluation criteria are common to
  all the programmes of the Sixth Framework
  Programme and are set out in the European
  Parliament and the Council Regulations on the
  Rules for Participation (Article 10). These are
• a) Scientific and technological excellence and
  the degree of innovation
• b) Ability to carry out the indirect action
  successfully and to ensure its efficient
  management, assessed in terms of resources and
  competences and including the organisational
  modalities foreseen by the participants
• c) Relevance to the objectives of the specific
  programme
• d) European added value, critical mass of
  resources mobilised and contribution to Community
  policies
• e) Quality of the plan for using and
  disseminating the knowledge, potential for
  promoting innovation, and clear plans for the
  management of intellectual property.

9
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• Furthermore, in applying paragraph (d) above, the
  following criteria are also to be taken into
  account
• a) For networks of excellence, the scope and
  degree of the effort to achieve integration and
  the networks capacity to promote excellence
  beyond its membership, as well as the prospects
  of the durable integration of their research
  capabilities and resources after the end of the
  period covered by the Communitys financial
  contribution
• b) For integrated projects, the scale of the
  ambition of the objectives and the capacity of
  the resources to make a significant contribution
  to reinforcing competitiveness or solving
  societal problems
• c) For integrated initiatives relating to
  infrastructure, the prospects of the initiatives
  continuing long term after the end of the period
  covered by the Communitys financial contribution.

10
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• Horizontal issues
• Are there gender issues associated with the
  subject of the proposal? If so, have they been
  adequately taken into account?
• Have the applicants identified the potential
  ethical and/or safety aspects of the proposed
  research regarding its objectives, the
  methodology and the possible implications of the
  results? If so, have they been adequately taken
  into account in the preparation of the proposal?
• To what extent does the proposal demonstrate a
  readiness to engage with actors beyond the
  research community and the public as a whole, to
  help spread awareness and knowledge and to
  explore the wider societal implications of the
  proposed work?
• Have the synergies with education at all levels
  been clearly set out?
• If third country participation is envisaged in
  the proposal, is it well justified and the
  participation well integrated in the activities?

11
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• 1. Relevance (threshold score 3 out of 5)
• The extent to which the proposed project
  addresses the objectives of the work programme.
• 2. Potential impact (threshold score 3 out of 5)
• The extent to which
• the proposed project is suitably ambitious in
  terms of its strategic impact on reinforcing
  competitiveness (including that of SMEs) or on
  solving societal problems.
• the innovation-related activities and
  exploitation and/or dissemination plans are
  adequate to ensure optimal use of the project
  results.
• the proposal demonstrates a clear added value in
  carrying out the work at European level and takes
  account of research activities at national level
  and under European initiatives (e.g. Eureka).
• 3. ST excellence (threshold score 4 out of 5)
• The extent to which
• the project has clearly defined objectives.
• the objectives represent clear progress beyond
  the current state-of-the-art.
• the proposed ST approach is likely to enable
  the project to achieve its objectives in research
  and innovation.

12
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• 4. Quality of the consortium (threshold score 3
  out of 5)
• The extent to which
• the participants collectively constitute a
  consortium of high quality.
• the participants are well-suited and committed
  to the tasks assigned to them.
• there is good complementarity between
  participants.
• the profiles of the participants, including those
  to be included later, have been clearly
  described.
• the real involvement of SMEs has been adequately
  addressed.
• 5. Quality of the management (threshold score 3
  out of 5)
• The extent to which
• the organisational structure is well matched to
  the complexity of the project and to the degree
  of integration required.
• the project management is demonstrably of high
  quality.
• there is a satisfactory plan for the management
  of knowledge, of intellectual property and of
  other innovation-related activities.
• 6. Mobilisation of resources (threshold score 3
  out of 5)
• The extent to which
• the project mobilises the minimum critical mass
  of resources (personnel, equipment, finance)
  necessary for success.
• the resources are convincingly integrated to
  form a coherent project.
• the overall financial plan for the project is
  adequate.
• Overall threshold score 24 out of 30.

13
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• 1. Relevance (threshold score 3 out of 5)
• The extent to which the proposed project
  addresses the objectives of the work programme.
• 2. Potential impact (threshold score 3 out of 5)
• The extent to which
• Europe has a strategic need to strengthen ST
  excellence on the topic by means of a
  restructuring of the existing research capacities
  and the way research is carried out.
• the goals of the network are, in that connection,
  suitably ambitious particularly, in terms of
  achieving European leadership and acting as a
  world force on this topic.
• the proposal demonstrates a clear added value in
  carrying out the work at European level and takes
  account of research activities at national level
  and under European initiatives (e.g. Eureka).
• there is an effective plan for spreading
  excellence, exploiting results and disseminating
  knowledge, including to SMEs and to those outside
  the network.
• the proposed approach is likely to have a durable
  structuring impact on European research.
• 3. Excellence of the participants (threshold
  score 3 out of 5)
• The extent to which
• the participants are currently conducting
  excellent research relevant to the topic of the
  network or are capable of important contributions
  to the joint programme of activities.
• the participants are well suited to the tasks
  assigned to them.
• they collectively have the necessary critical
  mass of expertise and resources to carry out the
  joint programme of activities successfully.

14
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• 4. Degree of integration and the joint programme
  of activities (threshold score 4 out of 5)
• The extent to which
• the expected degree of integration justifies
  supporting the proposal as a network of
  excellence.
• the joint programme of activities is sufficiently
  well designed to achieve the expected degree of
  integration.
• the participating organisations have made a
  convincing commitment towards a deep and durable
  integration continuing beyond the period of
  Community support.
• 5. Organisation and management (threshold score 3
  out of 5)
• The extent to which
• the organisational structure of the network
  provides a secure framework for any necessary
  structural decisions to be taken
• the management of the network is demonstrably of
  high quality.
• there is a well-considered plan for promoting
  gender equality in the network.
• Overall threshold score 20 out of 25.

15
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• 1. Relevance (threshold score 3 out of 5)
• The extent to which the proposed project
  addresses the objectives of the work programme.
• 2. ST excellence (threshold score 4 out of 5)
• The extent to which
• the project has clearly defined and well focused
  objectives.
• the objectives represent clear progress beyond
  the current state-of-the-art.
• the proposed ST approach is likely to enable the
  project to achieve its objectives in research and
  innovation
• 3. Potential impact (threshold score 3 out of 5)
• The extent to which
• the proposed project is likely to have an impact
  on reinforcing competitiveness or on solving
  societal problems.
• the proposal demonstrates a clear added value in
  carrying out the work at European level and takes
  account of research activities at national level
  and under European initiatives (e.g. Eureka).
• exploitation and/or dissemination plans are
  adequate to ensure optimal use of the project
  results.

16
I criteri ed i sottocriteri di valutazione per i
differenti strumenti di finanziamento

• 4. Quality of the consortium (threshold score 3
  out of 5)
• The extent to which
• the participants collectively constitute a
  consortium of high quality.
• the participants are well-suited and committed to
  the tasks assigned to them.
• there is good complementarity between
  participants.
• the opportunity of involving SMEs has been
  adequately addressed.
• 5. Quality of the management (threshold score 3
  out of 5)
• The extent to which
• the project management is demonstrably of high
  quality.
• there is a satisfactory plan for the management
  of knowledge, of intellectual property and of
  other innovation-related activities.
• 6. Mobilisation of resources (threshold score 3
  out of 5)
• The extent to which
• the project foresees the resources (personnel,
  equipment, financial) necessary for success.
• the resources are convincingly integrated to form
  a coherent project.
• the overall financial plan for the project is
  adequate.
• Overall threshold score 21 out of 30.

17
Il processo di selezione delle proposte

• Quality. Projects selected for funding must
  demonstrate a high scientific, technical and
  managerial quality in the context of the
  objectives of the RTD programme in question.
• Transparency. In order to provide a clear
  framework for researchers preparing proposals for
  funding and for evaluators evaluating proposals
  the process of reaching those funding decisions
  must be clearly described and available to any
  interested party. In addition, adequate feedback
  must be provided to proposers on the outcome of
  the evaluation of their proposals.
• Equality of treatment. A fundamental principle of
  EU RTD support is that all proposals should be
  treated alike, irrespective of where they
  originate or the identity of the proposers.

18
Il processo di selezione delle proposte

• Impartiality. All proposals are treated
  impartially on their merits.
• Efficiency and speed. The procedures have been
  designed to be as rapid as possible, commensurate
  with maintaining the quality of the evaluation,
  appropriate use of public money and respecting
  the legal framework within which the specific
  programme is managed.
• Ethical considerations. Any proposal which
  contravenes fundamental ethical principles may be
  excluded from being evaluated or selected at any
  time.

19
Il processo di selezione delle proposte

• Appointment of independent experts
• Calls for applications from individuals published
  in the Official Journal of the European
  Communities or
• Calls addressed to research institutions with a
  view to establishing lists of suitable
  candidates.
• The lists of individuals from which panels of
  experts may be chosen, are drawn up by the
  Commission using the following selection
  criteria
• an appropriate range of competencies
• an appropriate balance between academic and
  industrial expertise and users
• a reasonable gender balance
• a reasonable distribution of geographical origins
  of independent experts
• regular rotation of independent experts.
• I valutatori sono retribuiti dalla CE !

20
Il processo di selezione delle proposte

• Impegni sottoscritti dai valutatori
• Conflict of interest
• Confidentiality
• (Independent observers)
• Eventualmente ammessi al processo di selezione

21
Il processo di selezione delle proposte

• Step 1 Briefing of the independent experts
• Step 2 Individual evaluation of proposals
• Each proposal is evaluated against the applicable
  criteria independently by several experts who
  fill in individual evaluation forms giving marks
  and providing comments.
• Proposal marking
• 0 - the proposal fails to address the issue under
  examination or can not be judged against the
  criterion due to missing or incomplete
  information
• 1 - poor
• 2 - fair
• 3 - good
• 4 - very good
• 5 excellent
• Where appropriate, half marks may be given.
• The outcome of this step is the Individual
  Assessment Report

22
Il processo di selezione delle proposte

• Step 3 Consensus meeting
• For each proposal a consensus report is prepared.
  The report faithfully reflects the views of the
  independent experts referred to in Step 2.
• Nella (quasi) totalità dei casi il Consensus
  meeting termina con il consenso unanime dei
  valutatori. Il meeting è moderato da uno
  scientific officer della CE.
• The experts attempt to agree on a consensus mark
  for each of the blocks of criteria. They justify
  their marks with comments suitable for feedback
  to the proposal coordinator and agree on an
  overall consensus report, which is signed by
  them.
• The outcome of the consensus step is the
  consensus report

23
Il processo di selezione delle proposte

• Step 4 Panel evaluation
• A panel discussion may be convened, if necessary,
  to examine and compare the consensus reports and
  marks in a given area, to review the proposals
  with respect to each other and, in specific cases
  (e.g. equal scores) to make recommendations on a
  priority order and/or on possible clustering or
  combination of proposals. The panel discussion
  may include hearings with the proposers.
• A Commission official acts as moderator of the
  Panel
• Cross readings
• Comparison of consensus reports
• grouping of proposals
• The outcome of the Panel meeting is
• An evaluation summary report for each proposal
• A list of proposals passing thresholds, if any,
  along with a final mark for each proposal passing
  the thresholds and the panel recommendations for
  priority order.

24
Il processo di selezione delle proposte

• Hearings (Audizione)
• Per i nuovi strumenti del 6 PQ è prevista
  unaudizione individuale per ogni proposta che ha
  superato la soglia. Il panel di cui al punto
  precedente si riunisce (eventualmente in una
  composizione ridotta) ancora una volta (dopo un
  tempo opportuno, cioè circa un mese) e, al
  termine delle audizioni, propone la graduatoria
  di merito finale.
• Final steps
• Reporting on the evaluation process
• Feedback to the proposers
• Negotiation
• Selection of proposals
• These final steps do not involve the independent
  experts

25
Commenti sul processo di selezione

• Il mio personale commento è che il processo di
  selezione raggiunge tutti gli scopi che si
  prefigge ed è il più efficiente ed obiettivo tra
  i vari processi a cui ho partecipato come
  soggetto attivo e/o passivo.
• Alcuni feedback da valutatori IST
• It was fascinating to see how such a complex
  problem, namely how to rank such a large number
  of disparate proposals, could be solved, and it
  was my pleasure to participate in the process.
• I'm glad that finally we made it! It was hard
  working, as you say, but I really enjoyed it I
  enjoyed working with clever and skilled people as
  evaluators and very professionals people in the
  EU commission. I enjoyed the international
  involvement and the possibility of knowing
  interesting people in the field from all over
  Europe. I enjoyed the possibility of having an
  overview of top level research people is now
  working on.

26
Commenti sul processo di selezione

• I learned a lot of very important things about
  science, present and future trends, and mainly
  how a good proposal should be prepared and how it
  should not
• It was great experience to take part in EC
  Project evaluations for me. I have learned how to
  work hard, precisely and effectively and without
  any external pressures
• I wish to say that I was very glad to participate
  in evaluations of IST proposals. Not only because
  of the scientific topic which is extremely
  interesting for me but also because of perfect
  organisation, competence of your team and
  friendly atmosphere during evaluation. Thank you
  very much
• I have been already in Growth panels in the past
  and had the opportunity to realize that the
  commission do everything is possible to be fair
  and objective when evaluating proposals and
  distributing public money. I do believe that IST
  staff is the paradigm in that manner

27
Paragone con le procedure di valutazione di
alcuni altri programmi

• INTAS
• The International Association for the Promotion
  of Co-operation with Scientists from the
  New Independent States (NIS) of the Former Soviet
  Union
• COST
• Founded in 1971, COST is an intergovernmental
  framework for European CO-operation in the field
  of Scientific and Technical Research, allowing
  the co-ordination of nationally funded research
  on a European level. COST Actions cover basic and
  pre-competitive research as well as activities of
  public utility
• Running Actions 200 Total Budget (2005)
  13,500,000
• MIUR - PRIN

28
Grazie! ... e buon appetito!