Why You Can’t Trust Most AI Studies

August 18th. A new MIT report on generative AI comes along. Bottom line: 95% of generative AI pilots fail. Everyone shares it. “Generative AI is doomed,” says one. “I knew it! ChatGPT doesn’t even know how to add numbers reliably,” says another. “Humanity always wins in the end,” says a third.

October 28th. A new Wharton report on generative AI comes along. Bottom line: 75% of enterprises are already seeing positive ROI on generative AI. Everyone shares it. “This is a pretty big deal,” says one. “And it’s just the beginning!” says another. “If you’re not using AI, you’re not gonna make it…” says a third.

I read both reports. Both seem fine at first glance.

What’s going on?

The quality of information surrounding AI is awful

Before I go into those two reports, you should keep in mind three things:

1. Beware the man of one study: You’d be scared to know the number of research studies from mature disciplines (e.g., medicine or social psychology) that pass peer review, are published in prestigious journals, and even infiltrate popular culture (e.g., Milgram’s experiment on obedience and the marshmallow test on delayed gratification) that turn out not to be replicable or whose findings are heavily disputed. Or worse: studies that originally found the opposite of what’s actually true (that’s called a reversal).

2. AI is in wartime, which means narrative > truth: Every single thing you hear about AI at this time (including, let’s not be hypocrites, this very blog) is biased to some degree by the fact that AI is popular, there’s a lot of money at stake, and there’s a lot of uncertainty on how it will turn out. I try to be honest in my approach—I have zero money at stake besides what I make by writing, which is contingent on being honest!—but there’s no denying biases affect us all.

3. Extremes lead the conversation: Corollary of the above. If you are moderate in your takes about AI, a more daring (or shameless) person will eat your lunch simply by adding a bit of hyperbole. Then another will add a bit more and eat their lunch. And so on. The ultimate state of affairs (which you will realize is already happening) is that the loudest, most shameless people gather at the