ARS Peer Review Process Making scientific quality review work for you

1
ARS Peer Review Process Making scientific
quality review work for you
Ed Cleveland Scientific Quality Review Officer
(SQRO) Office of Scientific Quality Review (OSQR)

2
Creation of OSQR

• 1998 Farm Bill
• ARS research peer-
• reviewed every 5 years
• Most review panelists external to ARS
• Satisfactory review
• before beginning
• research

3
Congressional Mandate
Stakeholder Workshop
National Program Action Plan
Program Assessment
Input
Input
PDRAM
Plan
Assess
Project Plan Outline
Annual review
Implement
Research Project Plan
Peer Review
Research initiated
Certification
4
COOPERATION BETWEEN ALL LEVELS DURING PROJECT
PLAN PREPARATION PROCESS
EXTERNAL REVIEW
INTERNAL REVIEW
RESEARCH TEAM (individual scientists)
OSQR OUTSIDE REVIEW PANELS
Internal dialogue and cooperation is essential
AREA OFFICE NATIONAL PROGRAM STAFF
Review is a dialogue Panel Recommendations and
ARS Responses
5
Congressional Mandate
Stakeholder Workshop
National Program Action Plan
Program Assessment
Input
Input
PDRAM
Plan
Assess
Project Plan Outline
Annual review
Implement
Research Project Plan
Peer Review
Research initiated
Certification
6
Goal of Peer Review
Enhance research through independent, expert
examination of PROSPECTIVE plans for scientific
and technical merit. Not an evaluation of the
ARS, its mission, National Programs, project
budgets, or personnel management
OSQR
7
Context of Peer Review

• Research must be relevant to an ARS National
  Program Action Plan
• Primary driver is the need to solve a problem,
  not investigator curiosity or idea novelty
• Projects are not in competition for funding
• Evaluation generates an Action Class and
  recommendations for improving.
• Research plan must receive a passing Action Class
  in order to proceed.

OSQR
8
Peer Reviews

• Provide external review by peers of the quality
  of a prospective project plan
• Identify potential areas where impact of
  scientific effort can be increased
• Increase the awareness of the quality and extent
  of ARS research programs

9
What is Reviewed?

• Adequacy of Approach and Procedures
• Probability of Successfully Accomplishing the
  Projects Objectives
• Merit and Significance

OSQR
10
Action Classes
No revision Excellent, no change needed Minor
revision very good, a few modifications
required Moderate revision Good, but has some
important areas to address Major revision
required Requires significant changes or
additions Not feasible Major flaws or not
possible to assess
OSQR
11
Project Plan Review vs. Peer Review of an
Original Research Article
OSQR Review Like Review of a Paper for
Publication (strong advisory component) -
Editor SQRO - Two outcomes 1. Publish
after revision as monitored by the editor
(SQRO). Reviewers clear on what researchers are
planning (minor gaps in info). (no, minor,
moderate revision) 2. Publish after revision
and reexamination by both reviewers and SQRO.
Reviewers not at all clear about what
researchers are planning (major gaps in info).
(major revision, not feasible)
12
Frequent Panel Comments

• Hypotheses are poorly constructed and unclear
• Lack of appropriate detail in Approach, Procedure
  and Experimental Design
• Weak or no connection between project objectives
• Readability, narrative flow, rationale for
  research not developed
• Role of project team members, including
  collaborators, not well-defined
• Milestones and timelines vague
• Perfunctory Contingency Plans
• Project management and progress evaluation not
  documented

See Statistician
Provide Diagram
Tie Contingencies to Milestones
OSQR
13
Hypotheses

• Most research in ARS is hypothesis-driven. Make
  sure these are credible, scientifically testable
  (i.e., falsifiable) hypotheses related to the
  objectives.
• One of the most frequent comments OSQR receives
  from reviewers is that the plans do not contain
  real, testable, hypotheses.
• Get advice from Statistician.

14
What Is a Real Hypothesis? Definitions A
hypothesis is a tentative statement that proposes
a possible explanation to some phenomenon or
event. It is an assumption written in a clear,
concise manner about what you think will happen
in your project A hypothesis is a logical
supposition, a reasonable guess, an educated
conjecture
15
The value of a well constructed hypothesis is to
provide direction for your project, keep your
investigation focused, and forces one to think
about what results to look for in an experiment.
The development of a good hypothesis is not
always an easy task, but without it, you may
collect aimless data. Take the time to refine
your hypothesis so you collect pertinent data.
Remember the hypothesis keeps you a seeker of
pertinent knowledge. Debbie Boykin, Statistician
MSA
16
Hypothesis Problems

• Hypotheses that are too complex, i.e., these are
  statements with and and or that
  essentially make the hypothesis a compound
  hypothesis, rendering it very difficult if not
  impossible to really test and reject because part
  might be rejected and part might not.
• Wiggle words. A hypothesis with may or might
  or could cannot be rejected its true no
  matter what result you get.
• Misdirected hypotheses about the researchers
  themselves. These say things like Discovering
  the mechanism behind X will enable us to. This
  tests the abilities of the researchers to take
  information and do something with it. Instead,
  the hypothesis should focus on the experimental
  system itself.

17
More problems

• Hypotheses that are statements of the obvious, or
  are scientifically trivial. Disease results
  from expression of genes for virulence in the
  pathogen and genes for susceptibility in the
  host.
• Too global. Quantifying X will provide
  significant increases in income for the
  industry. Can any 5-year project plan in ARS
  really test this?

18
A Hypothesis is not always needed...
Some research is not hypothesis-driven. This is
acceptable. Examples are some types of
engineering work and model development (Even in
these, however, there may be a basis for
hypothesis testing, e.g., testing whether a
particular modification in a model provides a
quantifiable improvement in how well the model
predicts some real phenomenon). If stating a
hypothesis is not appropriate, be sure the goal
or target of the work is clear.
19
Frequent Panel Comments

• Hypotheses are poorly constructed and unclear
• Lack of appropriate detail in Approach, Procedure
  and Experimental Design
• Weak or no connection between project objectives
• Readability, narrative flow, rationale for
  research not developed
• Role of project team members, including
  collaborators, not well-defined
• Milestones and timelines vague
• Perfunctory Contingency Plans
• Project management and progress evaluation not
  documented

OSQR
20
What is a picture worth?
21
Cohesiveness
does the plan credibly describe a cohesive,
integrated project, or does it look stove piped
with respect to how the objectives and personnel
interact? It is important to describe a
multi-personnel project in which the work hangs
together into an integrated whole. Your plan
should reflect how the work all comes together to
accomplish the overall goals and objectives of
the project.
22
Frequent Panel Comments

• Hypotheses are poorly constructed and unclear
• Lack of appropriate detail in Approach, Procedure
  and Experimental Design
• Weak or no connection between project objectives
• Readability, narrative flow, rationale for
  research not developed
• Role of project team members, including
  collaborators, not well-defined
• Milestones and timelines vague
• Perfunctory Contingency Plans
• Project management and progress evaluation not
  documented

OSQR
23
(No Transcript)
24
Writing a Clear PlanA well-done plan presents
the take home message from its opening pages.

• What is the problem?
• Why is it important?
• Where are you going with it?
• How are you going to get there?
• And how will you know you have arrived?
• This should be in brief on the opening pages

25
Correct Grammar and Spelling are Importantbut
not enough
Be sure your plan presents a clear, logical, path
to successat the outset and through the
document. Have scientific peers outside your
project and Unit read the plan for
understandability.
26
Frequent Panel Comments

• Hypotheses are poorly constructed and unclear
• Lack of appropriate detail in Approach, Procedure
  and Experimental Design
• Weak or no connection between project objectives
• Readability, narrative flow, rationale for
  research not developed
• Role of project team members, including
  collaborators, not well-defined
• Milestones and timelines vague
• Perfunctory Contingency Plans
• Project management and progress evaluation not
  documented

OSQR
27
Milestones
A Milestone is a MARKER that allows you to
measure or assess your progress. Used in the
ancient world to gauge distance from Rome
Complete a database on . Determine the
accuracy and bounds of uncertainty of a
model. Complete all work for a paper
on.. Complete the second year of a two-year
experiment on.. Complete the laboratory
analyses for field samples collected last
summer Deliver data from resistance trials to a
breeder who will
28
Weak Milestones
Continue studies on Cannot tell what
threshold would determine success on
this Develop understanding of Understanding
is a fleeting goal easily overturned by new
information Plan a study that Planning is an
ongoing activity for all scientists. Initiate
experiment on Could be as simple as a dated
entry in a notebook.
29
Contingencies
One good approach to Contingencies is to link the
section explicitly with Milestones that you
specify in the Milestones table that comes later
in the Plan. The Milestone might be acquiring
either positive or negative data/results. If you
create good Milestones that serve as decision
points along the way, then Contingencies are the
decisions that come as a result of achieving
those Milestones.
30
Lead Scientist and Scientist
Roles and Responsibilities

• Responsible for plan development and
  implementation (Lead scientist)
• Evaluate and document progress through the
  five-year cycle
• Interface with stakeholders providing information
  on impacts
• Prepare research papers and summaries of findings

31
COOPERATION BETWEEN ALL LEVELS DURING PROJECT
PLAN PREPARATION PROCESS
EXTERNAL REVIEW
INTERNAL REVIEW
RESEARCH TEAM (individual scientists)
OSQR OUTSIDE REVIEW PANELS
Internal dialogue and cooperation is essential
AREA OFFICE NATIONAL PROGRAM STAFF
Review is a dialogue Panel Recommendations and
ARS Responses
32
Roles and Responsibilities
NPS--Program Direction (Dialogue, Coordination,
Synthesize Objectives)
Everyone has responsibility for quality of
Project Plans
Research Team
Ensure Quality, Science Input
Lead Scientist
Research Leader
Laboratory Director
Ensure Quality, Management Viewpoint
Area Leadership
OSQR Panel
33
Project Plans are Linked to Team and Individual
Performance
Impact of Science, RPES
Project Plan development forces thoughtful
attention to project planning (hypotheses,
experimental design, statistics, milestones and
contingencies) which can enhance research and
career success.
34
Another way to look at it

• You may enjoy a 30-year career with ARS.
• Over that time the government may support project
  research with 20-30 million.
• Six times in that career you will be asked what
  you are doing with the governments money.

35

• OSQR RESOURCES
• Training Focusing More on What to Look For During
  
• INTERNAL REVIEW to Increase Quality of Plans
• Training of NP Scientists after PDRAMs Issued
• New SY Training
• Leadership Training
• New Research Leader Training
• Stakeholder Workshop Training
• Area SY Training (SAA done, MWA future)
• Training on Web-sitenew items added regularly

36
Project Plans

• The foundation of ARS research
• Link to performance and impact of individual and
  team
• Reflect project teams scientific expertise

OSQR
37
The new OSQR Manual

• Shorter by 30-35 percent (main part gt25 pages)
• More guidance on writing and presentation
• Format not quite a rigid
• (we care more about readability than if you use
  Times Roman)
• PPO replaces prospectus
• Reflects lessons learned over the years
• Information on areas of concern to reviewers
  highlighted
• Formal Agency review will begin shortly.
• (copies will be reviewed by all Areas and NPS)
• Anticipate release by FY08 (October 1).

38
Foundation of Project Plan Development
39
Roles and Responsibilities

• Program Direction
• National Program Leaders
• Set the objectives
• (in dialogue with research team)
• Project Team
• Scientists
• Lead Scientists
• Research Leaders
• Prepare Project Plan
• Management
• Research Leaders
• Center/Laboratory Directors
• Area Directors
• Ensure quality

OSQR