Import AI 424: Facebook improves ads with RL; LLM and human brain similarities; and mental health and chatbots

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

The inner lives of LLMs increasingly map to the inner lives of humans:
…Neuroscience study provides yet more evidence that AI systems and human brains converge on similar ways of representing the world…
Language models (and large-scale generative models more broadly) tend towards having complex internal representations of the world which increasingly correspond to how we think humans represent the world, according to new research from the Freie Universitat Berlin, University of Osnabruck, Bernstein Center for Computational Neuroscience, University of Minnesota, and the University of Montreal.
"We explore the hypothesis that the human brain projects visual information from retinal inputs, via a series of hierarchical computations, into a high-level multidimensional space that can be approximated by LLM embeddings of scene captions," the authors write. "We demonstrate that the visual system may indeed converge, across various higher-level visual regions, towards representations that are aligned with LLM embeddings."

What they did: They studied the Natural Scenes Dataset (NSD), which records the fMRI data from human brain responses to viewing thousands of complex natural scenes taken from the Microsoft Common Objects in Context (COCO) image database. To look at the differences between LLMs and human brains they took the captions from the dataset and used a sentence encoder based on the transformer architecture to project these descriptions into the embedding space of a LLM. They then " correlated representational dissimilarity matrices (RDMs) constructed from LLM embeddings of the image captions with RDMs constructed from brain activity patterns obtained while participants viewed the corresponding natural scenes".

The results show a lot of similarity: "LLM embeddings are able to predict visually evoked brain responses across higher level visual areas in the ventral, lateral and parietal streams". In other words, LLM embeddings of scene captions successfully characterize brain activity evoked by viewing the natural scenes. "We suggest that LLM embeddings capture visually evoked brain activity by reflecting the statistical regularities of the world, learned through their extensive language training, in ways that align with sensory processing."

The simplest way to understand this:

• When the brain finds two images similar, the LLM also finds their captions similar.

• When the brain finds two images different, the LLM also finds their captions different.

Why this matters - internal representational complexity maps ...