STEM Update #20: NSF Reviewer Ratings, Interpreting "and"s in NSF names

Thursday, October 10, 2024


Context: In my role as division director of Information and Intelligent Systems (IIS) at NSF, I’m sending out a short message to the IIS mailing list on the Second Tuesday Every Month (STEM). This is the installment for October 2024, even closer to being on time than last month’s.


When proposals are reviewed at NSF, a bunch of letters get assigned. Here’s a list of the letters I’ve seen on proposals since coming to NSF: 

 

C, DNF, E, F, FIP, G, HC, HP, LC, LP, MP, NC, ND, NDP, NR, OP, P, R, RWR, and V .

 

That can be kind of overwhelming, so here’s a guide to interpreting what they mean. First, these letters actually come in 3 different flavors depending on the phase of the merit-review process in which they are assigned (see STEM #4 for an overview of the process): (1) ratings made by individual reviewers, (2) recommendations selected by panels, and (3) decisions on a proposal. In today’s message, I’ll look at the first category. More on the other categories in future messages.

 

For ratings assigned by individual reviewers, the ratings (from highest to lowest) are:

 

 

NSF allows split ratings (E/V, F/P, etc.), which are almost always used by reviewers to signal that a proposal falls between two of these ratings. (In principle, non-consecutive ratings can be combined with a slash, but I haven’t seen it in my time at NSF. I’m pretty sure I tried to do it myself once as a reviewer way back in the day.)

 

To give you a feel for how reviewers use these ratings, here are generic versions of actual summary justifications I’ve seen in submitted reviews:

 

 

These are just examples. Justifications come in many shapes and sizes. These ratings are fed into the next step of the process, which is typically a panel discussion. I’m planning to talk about the ratings used at that stage next month.

 

Here’s a puzzle that isn’t super deep, but I’ve found that the concepts it introduces are helpful for understanding some of the names used at NSF.

 

First, some denotational semantics of phrases. I am (oversimplifyingly) mapping a phrase that can have the word "and" to a set of phrases that doesn't. Here's how:

 

[[ x ]] = { x }, 

[[ X Y ]] = { x y | x in [[ X ]], y in [[ Y ]] }, 

[[ X and Y ]] = [[ X ]] U [[ Y ]] .

 

In words, I’m interpreting spaces as a concatenation operator and “and” as a union operator. To provide two simple examples,

 

             [[ macaroni and cheese ]] = { macaroni, cheese }

 

and        

 

[[ cream cheese ]] = { cream cheese } .

 

That is, “macaroni and cheese” is interpreted as two one-word terms, whereas “cream cheese” is interpreted as one two-word term.

 

Things start to get interesting when we combine multiword terms and “and”s in the same phrase. For example, a phrase like “almond butter and milk” is ambiguous and has two different interpretations depending on how we add parentheses. To wit:

 

[[ (almond butter) and milk ]] = { almond butter, milk } 

 

vs.

 

[[ almond (butter and milk) ]] = { almond butter, almond milk }.

 

Basically, the “space” operator can be distributed over the “and” operator and we can get two slightly different sets. I’ve seen examples of both of these interpretations of this particular phrase online so I think I’m on solid (and tasty) ground here. 

 

Some phrases are complicated but still have a clear interpretation. I ran across the two-”and” phrase “grilled peanut butter and jelly and banana sandwich”, which I think should be parenthesized and translated as:

 

[[ grilled ((((peanut butter) and jelly) and banana) sandwich) ]] = 

                             { grilled peanut butter sandwich, grilled jelly sandwich, grilled banana sandwich }

 

and not, as one perverse example, as:

 

[[ (grilled peanut (butter and jelly)) and (banana sandwich) ]] = 

                             { grilled peanut butter, grilled peanut jelly, banana sandwich } .

 

Bringing this idea to NSF, you can interpret the name of the IIS division “Information and Intelligent Systems” two different ways:

 

             [[ Information and (Intelligent Systems) ]] = { Information, Intelligent Systems } 

 

and        

 

[[ (Information and Intelligent) Systems ]] = { Information Systems, Intelligent Systems } .

 

I’m pretty sure the second is the intended meaning. Indeed, that structure for a division name is very common at NSF. Of the 26 "division of" units at NSF, ten have the form “X X” and 7 have the form “X and X X”. No other pattern appears more than twice. And, the “X and X X” divisions are all actually “(X and X) X”. NSF loves distributing spaces over “and”s.

 

Ok, here’s the puzzle part:

 

By adding parentheses, how many distinct interpretations are there for the double “and” name of IIS’s directorate, “Computer and Information Science and Engineering”? Which do you think is the intended meaning and why?

 

Until next time.

 

-Michael