I think the recent anthropic paper on global workspaces in LLMs gives new insight into the Chinese Room Puzzle. The most surprising part of the paper for me was that averaging Jacobians across contexts could produce a sensible result that would let you read out the word a particular token in particular layer would produce. What that means is that starting from some intermediate layer, there is a *fixed coordinate system for representing meaning* = semantics, tied to verbalizable English. It's not the case that in one context "spider" points in this direction and in a other context it points in a totally different direction. "Spider" points in the same direction across all contexts, and so survives averaging. So we can think of different layers of LLM as talking to each other in this common coordinate system, or language, to finally produce a sensible output.
I think that is a very different picture of what is happening inside an LLM from the one Searle presents, where, in the particular example he gives in the video here, you have the restaurant script symbols, the story symbols, question symbols, etc. and they're all just 'squiggles' and 'squaggles' that you, the computer whose ability to understand is being questioned, put together through arbitrary syntactic rules. It turns out the squiggles and squaggles are all aligned to the same semantic coordinate system, i.e., there's only squiggles! So basically, there is a translation manual hidden in the computer. The Anthropic paper is telling us that the man in the sea of Chinese symbols is not as semantically helpless as Searle makes him out to be.
In the human brain these semantic directions are further tied to patterns of sensory input: https://t.co/SRv0kL1MzJ