I was recently on a panel to discuss the impact of artificial intelligence on society. For my opening ten-minute speech, I sketched a big-picture, speculative story about how advanced AI will eat away at human interdependence. It’s intended for a general audience, so don’t expect much rigour, precision, or engagement with the academic literature. Still, it’s a concise, accurate summary of what I genuinely think.

When thinking about how advances in artificial intelligence are likely to change society in the coming years, people tend to suffer from two failures of imagination. They either fail to grapple with how transformative this technology will be, or they misrepresent the nature of this transformation by anthropomorphising the technology, treating advanced AI as an alien species that will threaten human survival.

I will suggest that AI’s most significant impacts will come not from superintelligent agents that threaten human survival, but from superintelligent tools that serve human interests too well.

The First Failure: Underestimating Transformative AI

Much of the popular and academic discourse about AI and its dangers focuses on how it might exacerbate existing risks and concerns: for example, concerning misinformation, privacy, or how decision-making systems perpetuate biases against marginalised groups. These are important conversations worthy of serious attention. But if they are the only conversations, they are too conservative. They treat AI as a technology that will be confined to existing social structures, maybe making current problems worse, but not fundamentally remaking reality.

But AI is not and has never been just another technology. From its beginnings as a serious science and engineering project in the mid-twentieth century, it has always had a revolutionary goal: to build machines that aren’t just as smart as the smartest human beings—that can match all of our most impressive intellectual and behavioural abilities—but that are far smarter. That are “ultraintelligent” or “superintelligent”.

However, even this framing undersells the revolutionary nature of this project, because it treats human minds as the benchmark. But humans are just one species of African ape. The space of possible intelligences is unimaginably vast, many of them associated with capacities we can’t imagine. We have barely begun to explore this space.

Some people doubt that superintelligent machines are possible. They think there is something special about human intelligence that couldn’t be replicated in machines, or that the engineering challenge is simply too hard.