Wednesday, June 18, 2025
Context: In my role as division director of Information and Intelligent Systems (IIS) at NSF, I’m sending out a short message to the IIS mailing list on the Second Tuesday Every Month (STEM). This is the installment for June 2025.
Hi all,
In this, most likely my penultimate update, I thought it would be nice to share some information about funding for large language model (LLM) research. As someone who has been involved with AI for decades, I’ve found the last few years to be very interesting. The term AI gets bandied about in some ways that are much more general than before (anything involving computers!), but also oddly narrowly (only chatbots!). Regardless, LLMs are an object of intense study and it can be confusing to know where to send a proposal that features research on LLMs. Program officers in IIS sent me their perspectives on this question. (Thanks to Eleni, Sylvia, Chris, and Dan.) Here’s my summary of the kind of work that falls into each of the three primary IIS clusters:
Robust Intelligence (RI): The main focus of the proposal / contribution is on understanding or improving LLMs for language processing; extending large foundation models to include multimodal data; streamlining inference to use less computation, data, or human feedback; addressing robustness, accuracy, or interpretability challenges in LLMs.
Human-Centered Computing (HCC): The main focus of the proposal / contribution is on studying LLMs for human-AI interaction and its impact on people including collaboration, creativity, education, and augmenting communication; looking at the impact of LLMs when used for support.
Information Integration and Informatics (III): The main focus of the proposal / contribution is on studying domain-specific LLMs, often multimodal; devising ways to make LLMs more reliable and explainable, even beyond reasoning models; using LLMs in a wide range of settings including health; creating instructional content for relevant stakeholders like surgeons or patients.
The thing to keep in mind is---what is the primary contribution of your work? What community would care most about it? (An alert reader would notice that there is some overlap in LLM topics between RI and III. That’s correct.)
If you want to get a sense of the kinds of work we’ve already funded, try a search like this:
https://www.nsf.gov/awardsearch/simpleSearchResult?queryText=%22large+language+models%22+AND+HCC
You can substitute HCC with other clusters (III, RI) or even the name of the division (IIS) to take different slices of this space.
Other IIS-led programs like FRR (robotics) use similar rules---if the LLM is playing a supporting role and the main emphasis of the proposal is on robotics, submit to FRR. For programs like Smart Health, if the advances in LLMs (in any of the areas above) are aimed at also improving health, then submit to Smart Health. Overall, submit to the program that matches the main emphasis of the proposed work. Research on LLMs is growing fast, but just because a proposal depends on an LLM doesn’t mean that it’s about language.
Next, I wanted to mention that the CAREER program will continue next year. There was some documentation circulating that some people took to mean that the program was eliminated. It hasn’t been. We’ve also heard that some people believe applying early is good. While it’s a great idea to get your proposal in before the last minute, there’s no particular advantage to getting it in a month or two early. IIS has an upcoming office hour on CAREERs and I encourage you to attend if you have questions.
As division director, one of my roles at the NSF is to serve as a final check on proposals being declined and proposals being awarded funding from my division. I’ve generally split duties up with my deputy division director, Wendy. (Oh! By the way, Wendy has been officially asked to serve as acting division director when I depart next month. I encourage you to send her a friendly thank you, but don’t spam her inbox. Maybe people with names starting with A can send their thank yous this month, Bs in July, Cs in August, etc.? A steady stream of gratitude and moral support is just what she’ll need…)
Anyway, I’ve kept track of the funding we’ve approved (and what I’ve declined, but let’s not talk about that) since I came on board and was kind of stunned to see that the number is over $600M. As I child of the 70s, I watched a lot of the Bionic Man and the idea that my division has spent the equivalent of 100 Steve Austins is mind blowing to me. I’m very proud of the research community who is turning these dollars into top notch science and the program officers (and panelists) who have done the vetting to ensure the money is allocated as best as we can manage.
Ok, one last puzzle before I go. Once a week, as part of my exercise-and-pop-music regimen, I make a note of the current top ten songs on the Billboard Hot 100 list. Then, when I jog on the treadmill, I listen to those songs in reverse order. Billboard’s list includes, along with each song's current ranking, a marking indicating whether the song went up, down, or stayed the same from the previous week. (The other two markings are for songs that didn't have a rank the previous week because they are brand new to the Hot 100 or are returning to the Hot 100.)
I started keeping a document with each week's top 10. In the last 100 weeks, there were sixteen times the top ten one week was a permutation of the previous week's top 10. Of these, 4 could be reconstructed uniquely using logical reasoning from the previous week's top ten and the current week's markings. (By “current week’s markings”, I mean that we know, for each of the top ten positions, whether the song at that position had gone up, down, or stayed the same from last week.)
Assuming this week's top ten is a uniform permutation of last week's top 10, what is the probability that this week's top ten can be reconstructed uniquely using logical reasoning from last week's top ten and the current week's markings? (For extra credit, provide a formula that works for the top n instead of just the top 10.)
Note that the probability for n=10 is quite a bit lower than my empirical estimate (4/16 or 25%), presumably because the ranking of a song in one week tends to be very close to its ranking in the previous week. (Don’t send me a chatbot-generated solution unless you are very confident it is correct. Sometimes they generate utter nonsense. Gemini called it “a fascinating puzzle that combines logic, combinatorics, and a dash of pop culture”, but I wonder if it’s just kissing up to me.)
The truth is out there.
Michael