Reviewer Training School Renovation, IDEA and Technology Grants Program Wednesday, November 14, 2001

1
PROPOSAL REVIEWER TRAINING ???? Florida Learn
and Serve 2007
May 2007
2
SESSION GOALS

• Participants will learn about the importance of
  a Request For Proposal (RFP) competitive
  proposal review and why it is an essential part
  of the grant funding process.
• Participants will gain an understanding of the
  roles and responsibilities of grant proposal
  reviewers.
• Participants will learn about the review process
  and be sufficiently prepared to serve as an
  informed, fair, and proficient grant proposal
  reviewer.

3
This Session includes.

• The Proposal Review Process
• Program Office Role and Functions
• Proposal Reviewer Roles and Functions
• Grants Training Development Role and
  Functions
• Proposal Reviewer Hints and Help

4
The Significance of the RFP in the Proposal
Developmentand Submission Process
5
Request for Proposal (RFP)
The RFP

• Is a Notice
• Released by the grantor agency
• Announcing an opportunity to compete for funds
• Includes criteria for eligibility, proposal
  structure, proposal content, and submission

6
An RFP Should

• Be comprised of clear, concise requirements
• Dictate the format for the proposal application
• Include both qualitative and quantitative
  outcomes
• Set scoring criteria for each section
• Be followed, just as you would follow any

Refer to the RFPs Florida Learn Serve Pilot
Projects and Florida Learn Serve Continuation
Projects
7
The release of an RFP starts the official
clock ticking for qualified entities interested
in submitting a competitive proposal
application.
Once all submissions are received (and meet
certain preliminary criteria), they must be
evaluated and scored (rated) by reviewers.
Proposals earning enough points to equal or
exceed the cut off score are considered
eligible and may be recommended for funding.
8
Why is this significant to the proposal reviewer?
9
Because.

• Proposals can only be as good as an RFP allows!
• The applicant has no control over the RFP
  design.
• The task of the proposal reviewer is to make
  sense of the proposal and score it as fairly
  and objectively as possible.

Therefore
If an RFP is disjointed and confusing, the
proposal may reflect the same issues! ...and a
disjointed proposal is much more challenging
for the reviewer to score.
10
Role of Program Administrator during Review
Process

• S/he is considered the Proposal Review Leader and
  will
• be on-call
• manage instruments and supplies, if necessary
• answer technical and procedural questions
• not comment on individual applications nor
  provide opinions related to same
• check completed score sheets for correct
  addition, completeness, adequate scoring
  comments, and legibility
• prior to formally ending the review process,
  provide all score sheets to Grants Training and
  Development (GTD) for review
• follow-up with individual reviewers as needed

11
Recommended Proposal Review Methodology

• The Program Administrator selects qualified
  volunteers to participate as the reviewers.
• A Reviewer Training and Orientation is
  convened, via a face-to-face session, a web-based
  event, or a conference call.
• Each reviewer is assigned a specific number of
  proposals to evaluate and score.
• The Conflict of Interest statement is signed by
  each reviewer.
• An agreed-upon deadline is set for completing the
  reviews. (All reviews are expected to be
  completed by the pre-selected date and time.)
• Reviewers score each proposal individually,
  without input from other reviewers.

12

• Upon completion of the individual work, reviewers
  return the completed score sheets to Joe Follman.
  
• The Program Administrator and Grants Training and
  Development (GTD) staff review all score sheets
  to ensure the scores add up and that all
  reviewer comments are legible, on target,
  constructive, precise, and provide adequate
  justification. Reviewers may be contacted for
  clarification or additional information.
• The Program Administrator conducts a debrief of
  the proposal review process. (This can be
  accomplished via an email, a conference call, a
  participant feedback form, etc.)
• Upon completion of the debrief, Joe Follman will
  send an email to all reviewers stating the review
  is complete and formally closed.
• The reviewers are asked to complete a Grant
  Proposal Review Process evaluation and send it,
  as an email attachment, to GrantsTrainingandDevelo
  pment_at_fldoe.org within three (3) working days
  after the review closes.

13
On the one hand, reviewers are not
involved in the actual funding
decisions.On the other, reviewer scores and
comments provide critical input to those making
the award recommendations.
The Role of reviewer is an IMPORTANT one.
14
Reviewers are expected to

• Refrain from possible conflicts of interest
• Apply a fairness standard to all
• Expect a comparable level of effort from each
  applicant
• Score solely on the written information in each
  application

• Provide expertise as reflected in constructive
  comments supporting each score not awarded
  the total possible points
• Score using only whole numbers (no factions,
  no decimals)
• Maintain confidentiality (today, tomorrow, and
  beyond)

15
CONFLICT OF INTEREST

• A conflict of interest occurs if (or when) a
  reviewer has a conflict that in any way might
  create a biased response.
• Reviewers must sign a Conflict of Interest
  Statement after ensuring all eligible proposals
  present no conflict of interest.

Each Reviewer must FAX, ASAP, the signed Conflict
of Interest Statement to Pat Golay FAX (850)
922-2928. At the end of the review, the signed
original must be returned in the metered
pre-addressed envelope provided to all Reviewers.
16
IMPORTANT INSTRUCTIONS FOR THE REVIEWERS

• READ all proposals through (before scoring any of
  them) to get a feel for the proposals
  individually and in comparison to each other.
• EVALUATE the quality of EACH response in EACH
  proposal, at the beginning of the scoring process
  by considering the following
• Is the response.
• Relevant and responsive to qualitative criteria?
  
• Comprehensive and well thought-out?
• An effective, logical, and realistic approach to
  the problem?

17

• RATE each applicable section by assigning a point
  score.
• ALWAYS JUSTIFY, with written comments, ALL
  ratings of less than full points.
• 5. Comments explaining why a narrative component
  received a Perfect Score are strongly
  recommend. Generic comments, such as Meets all
  criteria state the obvious and add no value.
• 6. Comments that support a professional
  assessment are important for several reasons and
  can include, but are not limited to concrete
  examples, professional constructive criticism,
  suggestions for improvement, highlighting noted
  strengths, and/or emphasizing challenges.

18
The score sheet contains guiding statements for
each criterion section
The guiding statements do not carry equal weight.
Rather, the statements are prompts to help the
reviewer understand what the proposal should
address.
Please do not divide the points available for the
section and allocate an even number for each
criterion.
To accurately reflect the reviewers impressions
and expertise, scoring decisions should be based
on an overall assessment of the entire question,
taking all parts into consideration.
19
Written Comments are ESSENTIAL!

• Comments (feedback) justify each score!
• If there are questions about scoring, reviewers
  comments should be sufficient to defend the
  scores without further explanation (Even a year
  or two after the review!).
• Write comments that clearly explain, to anyone
  reading them, why a particular score was given.
• Always try to provide concrete suggestions to
  guide applicants in future efforts.
• Comments should be positive, constructive versus
  critical, professional, helpful, and impartial.
• Strength-based comments are important to the
  applicant and help anyone else reading the scored
  application understand why the scores were given.
  

20
Reviewer Hints
If you can not locate a required item, it is
better to say that you cant find it than to say
it isnt there.Another method is to say the
response isnt thorough regarding . . .If an
answer is located other than in the section of
the narrative where it belongs, please make a
note and then credit the answer in the correct
section on the score sheet.
Refer to the Handout A Typical Source of Scoring
Criteria (EDGAR)
21
THE FINAL SCORING PROCESS

• Several methods may be used to finalize the
  reviewers scores
• As an example, with teams of five or seven
  reviewers, the highest and lowest scores may be
  dropped before averaging the remaining scores.
• With teams of three reviewers, large
  discrepancies in scores may be adjusted through
  discussion with the Program Administrator and
  Grants Training and Development staff.

22
Different Approaches
Some reviewers.
- Never met a proposal they liked - Never met a
proposal they did NOT like - Subtract points
from a maximum - Add points to a minimum (zero
based) - Have a predetermined, personal scale

Consistency and ability to support scoring
decisions must be fundamental to all approaches.
23
Should proposal reviewers be concerned about
these issues?

• Form?
• Grammar?
• Readability?
• Neatness?

What about.
Opinions can and do differ one acceptable
approach may be to suggest that reviewers
overlook these issues UNLESS the state of the
proposal brings into question the applicants
commitment to the project.
24
We cant all be EXPERTS in every area!

• Be mindful of the possible tendency to utilize
  the
• Benefit of the Doubt strategy, i.e. assigning a
  higher than usual score in topic areas where a
  reviewer is not entirely familiar

If a reviewer is aware of this happening, it is
recommended that s/he review that component again
(and possibly adjust some scores) to make sure
all scoring is as consistent as possible.
25
All comments and other notes should be entered
directly on the score sheet. If not,
reviewers are reminded to mail all handwritten
notes not included on the score sheet in the
metered pre-addressed envelope provided.
26
The RFP, the entire Grant proposal packet and all
documents and notes generated by reviewers during
the evaluation process are public documents.
Reviewers names, addresses and other personal
information are confidential and should only be
provided through Public Record requests.

• One of the reviewers tasks is to provide
  strength-based feedback intended to help the
  applicant when preparing future proposals.
• It is very important to make sure written
  comments are professional, clearly stated,
  constructive, and useful.
• Applicants are encouraged to obtain copies of
  completed scoring packets. Reviewer scores
  and comments provide feedback that may help
  the next time the applicant develops a proposal.

Reviewer comments count! Success rates improve
in the next competition among those who request
the review packet. CAUTION See H-7 Grant
Investigation
27
The Final Steps

• The Program Administrator will not close the
  review until all score sheets are assessed for
  accuracy, completeness, and quality and
  appropriateness of comments.
• All score sheets, other reviewer notes, and
  comments are routed to the Program
  Administrator, electronically or via mail.
  
• Following internal review and final approval
  of the score sheets by DOE, the reviewers
  are notified that they have
• completed their portion of the competitive
  process.
• The reviewers complete a Grant Proposal
  Review Process Evaluation form and email it
  to
• GrantsTrainingandDevelopment_at_fldoe.org.
• Reviewer feedback is IMPORTANT and
• helps to improve the RFP Competitive process.

28
We hope you complete the proposal reviewing
experience feeling positive about the
well-thought-out recommendations that you have
made
29
And that participating as a proposal reviewer
was a great learning experience for you, too!
30
Contact Information
Grants Training and Development Bureau of
Contracts, Grants, and Procurement Florida
Department of Education325 W. Gaines Street,
Room 344Tallahassee, Florida 32399-0400
Main [phone] SunCom 205-0499 Fax
[phone] SunCom 205-0730
Susan Busch Susan.Busch_at_fldoe.org Matthew
Caldwell Matthew.Caldwell_at_fldoe.org Susan
Howell Susan.Howell_at_fldoe.org
31
THANK YOU! Your valuable time and expertise are
greatly appreciated!