David Thorstad is one of the more interesting critics of effective altruism. Unlike some, his objections are consistently thoughtful and interesting, and he’s against malaria, rather than against efforts to do something about it. Thorstad wrote a paper titled Three mistakes in the moral mathematics of existential risk, in which he argues that when one corrects for a few errors, the case for existential threat reduction becomes a lot shakeier. I disagree, and so I thought it would be worth responding to his paper.

The basic argument that Thorstad is addressing is pretty simple. The future could have a very large number of people living awesome lives (e.g. 10^52 according to one estimate). However, if we go extinct then it won’t have any people. Thus, reducing risks of extinction by even a small amount increases the number of well-off people in the far future by a staggering amount, swamping all other values in terms of utility.

1 Cumulative and background risk

The first error Thorstad discusses is neglect for cumulative risk. The standard explanation of why the future will have extremely huge numbers of people is that it could last a very long time. But it lasting a very long time means that there are many more opportunities for it to be destroyed.

In a billion years, there are a million 1,000 year times slices. So even if the odds of going extinct per thousand years were .001%, the odds we’d survive for a billion years would only be 0.004539765%. Actions that lower short-term existential risks by a little bit aren’t hugely significant because the odds of them influencing whether humans survive for a very long time are low.

Thorstad similarly claims that models of existential risk reduction don’t take into account background risk odds—the odds that we’d go extinct from something else if not this. If you think existential risks are significant this century, which you have to in order to think that it’s worth working on them, then probably they’re significant in other centuries, so we’re doomed anyways!

I think there are two crucial problems with this.

The first and biggest one: you should assign non-trivial credence to humans reaching a state where the odds of extinction per century are approximately zero. The odds are not trivial that if we get very advanced AI, we’ll basically eliminate any possibility of human extinction ...