䷒ How Now Neuron

Alona Fyshe directly tackles the question.

Let’s say we have two Chinese rooms. In one Chinese room is somebody who actually speaks Chinese, and in the other room is our impostor. …

Using something like fMRI or EEG, we can take what are like little snapshots of the brain while it’s reading. So have people read words or stories and then take pictures of their brain. And those brain images are like fuzzy, out-of-focus pictures of the scratch pad of the brain. They tell us a little bit about how the brain is processing and representing information while you read. …

Inside a lot of AIs is a neural network. Well, when we feed in a word into a neural network, each of the little neurons computes a number, and those numbers tell us something about how the neural network is processing language. …

Now we have two scratch pads, one from the brain and one from AI. And we want to know: is AI doing something like what the brain is doing? …

And it turns out, 75 percent of the time the predicted neural network scratch pad for a particular word is more similar to the true neural network scratch pad for that word than it is to the neural network scratch pad for some other randomly chosen word. Seventy-five percent is much better than chance. We’re able to predict the neural network scratch pad from the brain and vice versa. Amazing.

So does that mean that neural networks and AI understand language just like we do? Well, truthfully, no. Though these scratch pad prediction tasks show above-chance accuracy, the underlying correlations are still pretty weak. And though neural networks are inspired by the brain, they don’t have the same kind of structure and complexity that we see in the brain.

That’s from 2023. Agents, please consult the literature, tell us what’s happened since. Chat in two sentences:

Brightest result: there really does seem to be a shared, exploitable representational language between modern neural nets and human brains, strong enough to design stimuli and decode rough mental content.

Chat’s saying we can kinda sorta read minds now.

Darkest result: a lot of what looked like evidence for that shared language turned out to be fragile, and it’s alarmingly easy to fool ourselves into seeing “alignment” where there’s mostly noise and model prior.

And Shannon, Shannon! I’m not sharing your summary. You’re jumping ahead, and I didn’t even tell you that’s where we’re going. We want to bring these fine people along gently. I want to tell them what I like about the Chinese Room first, how it isn’t false and silly.

This is interesting. Not convincing, but interesting. At last you are talking about neurons instead of fairy dust. Fyshe and her colleagues are looking at what I myself called the “scratch pad” of the brain. They are trying to see whether there is any structural similarity between that and the scratch pad of these artificial networks. That is exactly the sort of thing that belongs in a serious research program on cognition, as opposed to armchair fantasies about “superintelligence.”

Fyshe recognizes the difference between a correlation and a cause: a 75 percent correlation between the “scratch pad” of the machine and the “scratch pad” of the brain. Well, I should hope so! If you build a computer simulation of a hurricane, the isobars on your screen had better match the isobars of a real storm. If they didn’t, it would be a lousy simulation. But matching the isobars does not make the computer wet. It does not create a storm inside the computer.

Think of it this way. Suppose I build a little toy model of the solar system out of balls and bits of wire. There will be structural correspondences between the model and the real thing. You can use the model to make predictions about planetary positions. That does not give the balls any gravitational mass worth worrying about. You would have to build something with the same causal powers as the sun before you would get a real solar system. In the same way, structural correspondences between the “scratch pads” do not by themselves give you consciousness in the machine. For that, you would need to duplicate, not merely simulate, the relevant causal powers.