Tuesday, November 12, 2024
Context: In my role as division director of Information and Intelligent Systems (IIS) at NSF, I’m sending out a short message to the IIS mailing list on the Second Tuesday Every Month (STEM). This is the installment for November 2024, on time!
In STEM #20, I pointed out that reviewed proposals at NSF get lots of letters assigned to them. I mentioned having seen:
C, DNF, E, F, FIP, G, HC, HP, LC, LP, MP, NC, ND, NDP, NR, OP, P, RWR, and V .
I pointed out that these letters actually come in 3 different flavors depending on the phase of the merit-review process in which they are: (1) ratings made by individual reviewers, (2) recommendations selected by panels, and (3) decisions on a proposal. I then talked a bit about the first category to cover E, V, G, F, and P, along with some combinations thereof. These are ratings assigned by individual reviewers. Today, I’ll go through the other two categories, which should cover the remaining letters in the list.
Typically, the individual reviewers get together as a “panel” to discuss the proposal and the reviews and to make a joint recommendation. In IIS, we typically code the recommendations as Highly Competitive (HC), Competitive (C), Low Competitive (LC), and Not Competitive (NC). Former CISE program officer Jonathan Sprinkle proposed explaining these categories using an HGTV scheme. I had thought that stood for “high good to very good”, but it actually refers to the Home & Garden Television network, where you can watch shows about home improvement and real estate. In these terms, the ratings can be translated as:
HC: Move-in quality with almost no remodeling needed beyond minor things like repainting a room.
C: Very livable in current condition, but will want to remodel one or more rooms at some point.
LC: House has potential for great livability. However, will want to do some significant remodeling before you move in.
NC: Perhaps a tear down or complete gut of the structure is needed.
Another possible rating is Not Discussed in Panel (NDP). This designation is used when there is significant agreement about a weaker proposal: The rankings are consistent and there are common concerns in the reviews that make discussion unnecessary. People at NSF call this designation “triage”, and I’ve been haughtily telling them that they are using the word improperly. I thought “triage” meant sorting into three categories (hence the “tri-”): (1) doesn’t need to be discussed because it’s clearly over the bar for funding, (2) doesn’t need to be discussed because it’s clearly below the bar for funding, (3) needs discussion to decide on a recommendation. Since NDP is category 2 and everything else in category 3, it sounds to me like biage (two categories). Well, it turns out that the word triage comes from French (sure…), but from the word trier ‘separate out’ and is related to the word “try” (attempt), not “tri-” (three). Time for an apology tour…
To give you a feel for how panels use these ratings, here are generic versions of actual summary justifications I’ve seen in submitted panel summaries:
HC: All reviewers felt the proposal was well-written and novel in terms of intellectual merit. Although two reviewers felt the outreach activities were commonplace and lacking in detail, all reviewers agreed that the broader impact to the scientific community is strong.
C: The key strength is that the overall research question is important and timely, and the team has sufficient background and preliminary results to make a convincing case that they are the people to do it. Primary weaknesses identified revolved around whether the scope and content of the three research thrusts was well designed and work together; there was some disagreement on the panel on this question, but no weaknesses were identified as critical.
LC: Overall, the proposal is a good start, but too many details are vague, too broad, or missing. Some of the technical choices lack solid justification. Too much of the research tasks body is spent describing prior work and not enough describes proposed research. Due to the lack of specificity and novelty, the panel ranks this proposal as Low Competitive.
NC: This is a very bold proposal that might have broad impacts. Given the boldness of the proposal, the panelists wanted to see stronger planning related to collaborations with the industry groups and ways of guarding against negative consequences.
Note that no “slashies” (HC/C, say) are allowed at the level of a panel rating, so sometimes borderline proposals take a lot of discussion before they are assigned a rating the panel can agree on. Also, these ratings are intended to be absolute, not relative to the other proposals on the panel. Panels often rank the proposals relative to one another as well. That provides guidance to the program officer who ultimately makes the recommendation about funding, factoring in the overall funding portfolio, geographic diversity, and other concerns.
Some programs (including some managed by IIS) use other schemes. I’ve seen:
FIP (Fund If Possible), NR (Not Recommended for Funding by Panel)
F (Fund), FIP (Fund If Possible), DNF (Do Not Fund)
HP (High Priority), MP (Medium Priority), LP (Low Priority), ND (Not Discussed)
OP (Outsdanding Priority), HP (High Priority), MP (Medium Priority), LP (Low Priority), ND (Not Discussed)
(I’m pretty sure they meant “Outstanding”, though.)
In terms of a final decision for a proposal, NSF uses “award” and “decline” and doesn’t typically abbreviate them. However, proposals that don’t adhere to the published rules are marked “RWR” for Returned Without Review.
An alert reader might have caught that “F” can be mean “Fair” (probably not going to be funded) or “Fund” (probably going to be funded). While that sounds like a recipe for confusion, the letters are used at two different stages and so there’s really no ambiguity. So those are the assigned letters as I undersdand them!
Last month’s puzzle was: By adding parentheses, how many distinct interpretations are there for the double-”and” name of IIS’s directorate, “Computer and Information Science and Engineering”? Which do you think is the intended meaning and why?
Astute readers would recognize that counting the ways of adding parentheses can be seen as the problem of enumerating binary trees on 4 leaves. There are five distinct trees, corresponding to the following interpretations:
[[ ((Computer and Information) Science) and Engineering ]] = { Computer Science, Information Science, Engineering } .
[[ (Computer and (Information Science)) and Engineering ]] = { Computer, Information Science, Engineering } .
[[ (Computer and Information) (Science and Engineering) ]] = { Computer Science, Information Science, Computer Engineering, Information Engineering } .
[[ Computer and ((Information Science) and Engineering) ]] = { Computer, Information Science, Engineering } .
[[ Computer and (Information (Science and Engineering)) ]] = { Computer, Information Science, Information Engineering } .
Some observations. First, although line 2 and line 4 correspond to two different ways of adding parentheses, they have the same “meaning” (in the denotational semantics I proposed). So, the answer to the first question is that there are 4 distinct interpretations.
Next, note that lines 1 and 2 (and 3) result in interpretations where CISE covers the topic of “engineering”. But there is already an Engineering directorate (ENG), so that is unlikely to be the intended meaning. In addition, lines 2 (and 3) and 5 result in interpretations where CISE covers the topic of “computer”. That seems too narrow to be the intended meaning. So, with lines 1, 2, 4, and 5 ruled out, the most likely interpretation is line 3. On the positive side, that interpretation suggests CISE covers computer science (great!), information science (excellent!), computer engineering (outstanding!), and information engineering (is that a thing?). An informal poll of current CISE leadership suggests that, among those who have thought about it at all, “information engineering” isn’t an intentional part of the name. But I checked in with the previous Deputy Assistant Director of CISE and he’s always thought that was there on purpose. In fact, I’m willing to bet that there are a few people on this list in departments with “information engineering” in their name. (It seems be more common outside of the US, but I see at least one prominent example inside the US.) Wikipedia says "Information engineering is the engineering discipline that deals with the generation, distribution, analysis, and use of information, data, and knowledge in systems." So, I’m feeling pretty good about line 3!
Before moving on from this topic, I want to mention a new favorite example of mine within NSF: “Quantum Computing and Information Science”. It’s the only NSF name I’ve seen that has the form “X (X and X) X”, leading to the denotation { Quantum Computing Science, Quantum Information Science }. Very slick!
Until next time.
-Michael