Thursday, October 10, 2024
Context: In my role as division director of Information and Intelligent Systems (IIS) at NSF, I’m sending out a short message to the IIS mailing list on the Second Tuesday Every Month (STEM). This is the installment for October 2024, even closer to being on time than last month’s.
When proposals are reviewed at NSF, a bunch of letters get assigned. Here’s a list of the letters I’ve seen on proposals since coming to NSF:
C, DNF, E, F, FIP, G, HC, HP, LC, LP, MP, NC, ND, NDP, NR, OP, P, R, RWR, and V .
That can be kind of overwhelming, so here’s a guide to interpreting what they mean. First, these letters actually come in 3 different flavors depending on the phase of the merit-review process in which they are assigned (see STEM #4 for an overview of the process): (1) ratings made by individual reviewers, (2) recommendations selected by panels, and (3) decisions on a proposal. In today’s message, I’ll look at the first category. More on the other categories in future messages.
For ratings assigned by individual reviewers, the ratings (from highest to lowest) are:
E – Excellent
V – Very good
G – Good
F – Fair
P – Poor
NSF allows split ratings (E/V, F/P, etc.), which are almost always used by reviewers to signal that a proposal falls between two of these ratings. (In principle, non-consecutive ratings can be combined with a slash, but I haven’t seen it in my time at NSF. I’m pretty sure I tried to do it myself once as a reviewer way back in the day.)
To give you a feel for how reviewers use these ratings, here are generic versions of actual summary justifications I’ve seen in submitted reviews:
E: Overall, this is a well-organized, thought out and reasoned proposal. The research activities are well scoped, interconnected, and designed as well. The outreach and educational goals are also very well planned, and the project has great partnerships to help reach the research goals as well.
E/V: Overall the proposal is ambitious, and technically deep. The different thrusts are well connected, and the PI is well qualified to carry out the proposed research. At times it glances over some of the difficulties of the problem and does not sufficiently discuss its risks.
V: Overall, this is an exciting proposal that addresses a complex sociotechnical challenge. I would have liked to see more justification on the connections and ordering of the proposal research activities along with a more clear description of the dissemination plan for the proposed framework.
V/G: The proposal is strong and addresses a very important problem. The PI has a good track record in this area, and the research plan is reasonable.
G: Overall, this is a promising proposal from a highly qualified PI. The problems attacked by the project are important and the project results may bring clear impacts to many applications. On the other hand, some technical issues exist in the current research plan and may need careful resolution. The PI's experience working with undergraduate students is impressive.
G/F: Overall, the research problem and broader impacts of the proposal are great, but the technical aspects are lacking. The proposed research seems narrow and not sufficiently differentiated from published work.
F: Overall, the introduction is good, and the problem seems interesting. The technical aspects, motivation, and broader impacts fall short of expectations.
F/P: The proposal has some interesting ideas, but its transformative potential is unclear and lacks sufficient innovation.
P: The project is well motivated, but the proposed technical approaches are rather straightforward and lack technological depth and breakthrough potential.
These are just examples. Justifications come in many shapes and sizes. These ratings are fed into the next step of the process, which is typically a panel discussion. I’m planning to talk about the ratings used at that stage next month.
Here’s a puzzle that isn’t super deep, but I’ve found that the concepts it introduces are helpful for understanding some of the names used at NSF.
First, some denotational semantics of phrases. I am (oversimplifyingly) mapping a phrase that can have the word "and" to a set of phrases that doesn't. Here's how:
[[ x ]] = { x },
[[ X Y ]] = { x y | x in [[ X ]], y in [[ Y ]] },
[[ X and Y ]] = [[ X ]] U [[ Y ]] .
In words, I’m interpreting spaces as a concatenation operator and “and” as a union operator. To provide two simple examples,
[[ macaroni and cheese ]] = { macaroni, cheese }
and
[[ cream cheese ]] = { cream cheese } .
That is, “macaroni and cheese” is interpreted as two one-word terms, whereas “cream cheese” is interpreted as one two-word term.
Things start to get interesting when we combine multiword terms and “and”s in the same phrase. For example, a phrase like “almond butter and milk” is ambiguous and has two different interpretations depending on how we add parentheses. To wit:
[[ (almond butter) and milk ]] = { almond butter, milk }
vs.
[[ almond (butter and milk) ]] = { almond butter, almond milk }.
Basically, the “space” operator can be distributed over the “and” operator and we can get two slightly different sets. I’ve seen examples of both of these interpretations of this particular phrase online so I think I’m on solid (and tasty) ground here.
Some phrases are complicated but still have a clear interpretation. I ran across the two-”and” phrase “grilled peanut butter and jelly and banana sandwich”, which I think should be parenthesized and translated as:
[[ grilled ((((peanut butter) and jelly) and banana) sandwich) ]] =
{ grilled peanut butter sandwich, grilled jelly sandwich, grilled banana sandwich }
and not, as one perverse example, as:
[[ (grilled peanut (butter and jelly)) and (banana sandwich) ]] =
{ grilled peanut butter, grilled peanut jelly, banana sandwich } .
Bringing this idea to NSF, you can interpret the name of the IIS division “Information and Intelligent Systems” two different ways:
[[ Information and (Intelligent Systems) ]] = { Information, Intelligent Systems }
and
[[ (Information and Intelligent) Systems ]] = { Information Systems, Intelligent Systems } .
I’m pretty sure the second is the intended meaning. Indeed, that structure for a division name is very common at NSF. Of the 26 "division of" units at NSF, ten have the form “X X” and 7 have the form “X and X X”. No other pattern appears more than twice. And, the “X and X X” divisions are all actually “(X and X) X”. NSF loves distributing spaces over “and”s.
Ok, here’s the puzzle part:
By adding parentheses, how many distinct interpretations are there for the double “and” name of IIS’s directorate, “Computer and Information Science and Engineering”? Which do you think is the intended meaning and why?
Until next time.
-Michael