1. Background information:

When proposals come into the Economics Program at NSF, they are sorted by field, then distributed to one of three program officers. Proposals are reviewed in two ways: by members of the Economics Advisory Panel (there are 14) and by external reviewers. Each proposal is assigned to the two panelists whose research interests are closest to the proposal. The Panel meets about three months after the proposal deadline to discuss and rank all of the proposals. In addition, each proposal is sent to about 6 external reviewers for written reviews. (Proposals that are interdisciplinary or complex may be sent to more). Program officers do their best to assign proposals to reviewers with expertise in the subject of the proposal. Often, an external reviewer will be someone who has been suggested by the Principal Investigator (PI).

After the panel meeting, the program officers meet and make decisions about the proposals. They consider both panel rankings and external reviews in making their decisions.

Timing is critical for these proposals -- please consider putting them ahead of other papers in your queue of refereeing. Your review is most useful if it reaches the program officer before the panel meeting, which usually occurs about 6 weeks or so after you receive the request. That way the panel can consider your opinion as well as their own. If it arrives shortly after the panel meeting, it still can affect the decision. If your review arrives late, it will be irrelevant for the decision on the proposal (though it still will be sent to the PI).

2. What's in a good review?

Overall Length: The review should add up to about 1-2 pages unless you are feeling expansive or use a large font, in which case feel free to write more. The three-line review is not very helpful to anyone, though it is easy to understand the temptation to be terse. Really extensive comments are useful to the PI, but probably will not affect the decision, unless the issue you raise is complex. Keep in mind that for most PIs the research is just getting underway and your suggestions can do a great deal to shape the direction of that research. I recommend three paragraphs or sections:

First paragraph: What is the proposal about? What is its methodology? What literature or field of research does it fit into?

Second paragraph: Tell the program officer about the PI. Has he/she published well? Has she been productive under prior National Science Foundation support? Is this a "promising young investigator"? An "interdisciplinary collaborative team"? Anything else we should know?

Third paragraph: Evaluate the proposed project. Be constructive. Is the topic important? Is it well-motivated? What contribution will it make? Is there a clear research plan? If the proposal is not good, please try to spell out what would be necessary to make it good.

3. Things to remember:

First, remember that your review will go to the PI. Avoid the temptation to be nasty, even if the proposal is really bad, and the message you want to send is: Don't ever darken the door of NSF again. PIs who receive nasty reviews sometimes come back to haunt your friendly and overworked program officer. (This is a government agency, after all.) Your program officer has to write up an evaluation of this proposal based on the Panel discussion and external reviews. He/she may not know much about the field in which the PI is writing, and the discussion during the Panel meeting may be too brief for her to get a handle on it. Give her a hand by providing as much detail as you can about what is right/wrong with the proposal.

Try to avoid seeing the proposals in black and white (Excellent and Poor). Intermediate gray evaluations are very helpful to your program officer in making decisions at the margin. Nearly all of proposals that have consistent Excellent evaluations are funded. Many proposals with consistent reviews in the Very Good range are funded as well. On occasion proposals with mixed reviews will be funded, particularly those that are somewhat risky but seem promising for advancing the discipline. The information you provide is critical in choosing among the many Very Good and Good proposals.

Remember that this is not a journal. The funding rate on proposals is about 25% -- not the 5-10% of most top journals. While the proposal may not be perfect, try to determine whether the project, as you understand it, is worth funding. While a recommendation to "revise and resubmit" is probably the most common outcome of a journal submission, this is relatively rare for proposals. The reason for this is that a revision will go back to a different panel -- one that includes at least 1/4 new members. It is likely to go to at least three new external reviewers as well. It is not like dealing with a stable editor and set of referees.

There is a tendency for reviewers to fall prey to two biases. Try not to under-evaluate the proposals in the areas that you know best (also known as eating your children). The temptation is great to really nail the shortcomings of proposals that a reviewer knows the most about. The shortcomings of these proposals are, after all, easiest to see. By contrast, proposals in other fields can look very appealing - intriguing, nice puzzles, cute ideas, with shortcomings that are less apparent. But it is worth considering that your field is competing with all the other fields for funding. Reviewers in some fields are much nastier on average than reviewers in other fields. While program officers make a great effort to take this into account, their flexibility is limited by really negative reviews. On the other hand, try also not to over-evaluate your field relative to others. Here's the other bias. Reviewers sometimes succumb to the temptation to act as advocates for their own fields at the expense of everything that is not theirs (also known as touting your children). This skews the evaluation scores and makes accurate comparisons difficult. Balance is necessary.

The distribution of reviews is typically center-weighted, with a mean somewhere between Good and Very Good. Most panelists will have distributions that look like this, too. For external reviewers, it is useful to keep this distribution in mind. One or two ratings of "Good" are generally sufficient to ensure a proposal will not be funded. It is unusual for proposals with ANY ratings of "Fair" or "Poor" to be funded.

So as this cycle's proposals reach you, keep in mind how important these reviews are in determining the allocation of funding my the Economics Program. Your review can have a significant impact on the decisions that are made.