1 Forethought

You hear a lot of talk about AI alignment, the process of getting AI’s aims to line up with our own. This makes sense given that we’re currently in the process of building AIs that could be vastly smarter than people, and we don’t yet have a great plan for getting them to do what we want. Surprisingly, there isn’t as much talk about other kinds of AI preparedness. If we’re creating superintelligence, this will have a profoundly transformative impact on every aspect of our world, and it’s best to prepare.

Enter Forethought. They’re a research organization trying to figure out how to navigate a world of profoundly transformative AI. They have some extremely impressive people on the team, including Will MacAskill (one of the cofounders of effective altruism), Tom Davidson (formerly a senior research fellow at Open Philanthropy), and many others. Hopefully they will also hire me!

A while ago, Forethought published a report called Preparing for the Intelligence Explosion. I thought the report was thoughtful, cogent, and persuasive. Its core thesis is probably right, and it has huge implications if it is right. For this reason, it seemed like a good idea to summarize it. That’s what I’ll do here. I won’t say, every sentence, “the Forethought team says,” but just know, I’m primarily summarizing what they say, not giving my own thoughts.

Their thesis in a nutshell: AI is likely to radically transform the world, prompting staggeringly fast economic growth, many new technologies, and a number of unprecedented challenges. And the world isn’t ready.

2 The intelligence explosion

TLDR: AI is advancing very rapidly and shows no sign of stopping. Among other things, the number of AI models we can run is increasing 25x per year, 500X as fast as growth in number of human researchers. If this continues, we will get extremely rapid growth in research abilities, which will prompt unprecedented innovation—potentially hundreds of years of growth in a few years even on conservative assumptions that predict pretty dramatic slowdowns in AI capabilities growth. Advancements in research capabilities will lead to both a technological and industrial explosion.

The fine print…

Every year, human research capacity grows about 5% per year. In contrast the number of AI researchers we can create has been growing 25x per year, over 500 times as quickly. Research is the primary bottleneck to long-term ...