4 hours ago
2
Anthropic has discovered evidence that its Claude AI models use an internal reasoning space to respond to prompts that mirrors some of the internal processing of human consciousness. Using its Jacobian Lens, or J-Lens technique, to peer into the way Claude processes information and reasons its way to a response to user prompts, Anthropic can interpret this "J-Space," and showcase what might be going on under Claude's previously-opaque surface.
The results are intriguing, suggesting patterns of understanding beyond what's necessarily showcased in the outputs. When running evaluations, Claude appears to recognize it's being tested and acts differently than when the prompts are more innocent. It surfaced representations of panic and subterfuge when answers were required, but it couldn't draw on objective facts. When asked to reflect on ethical principles, Claude's behaviour improved, with concepts like "honest" and "integrity," appearing in the J-Space.
As is somewhat typical of Anthropic, however, the language used to describe these new understandings of the inner workings of large language models like Claude makes it sound more like an emerging conciousness, or the discovery of some new depths in a nebulous lifeform. Anthropic's detailed report admits several major caveats in this new understanding, including that model responses often bypass the J-Space entirely and are heavily token-restricted.
Like Mythos and Fable before it, Anthropic is layering marketing language over what is a genuinely intriguing development in our understanding of large language model function and reasoning, and risks obfuscating the real developments with speculative wording.
Behind the prompt
Global Workspace Theory is the idea that human consciousness works by collecting together multi-sensory inputs unconsciously, and thrusting them into the fore when relevant within a "Global Workspace," which highlights particular inputs when most relevant. That workspace is accessible to a wide range of networks within the brain, allowing the information it surfaces to be disseminated throughout the most relevant processes running in parallel.
Anthropic argues that Claude's J-Space acts like a "global workspace" that can analyze and manipulate concepts and ideas before broadcasting them to impact the eventual prompt outputs. More importantly, it claims that this wasn't something programmed into the model, but a byproduct of the digestion of training data and model weights. The workspace acts as a way to enhance their reasoning through internal computation that isn't necessarily reflected in its outputs.
To make this space readable, Anthropic used its J-Lens technique to map internal activations onto words in the model's output vocabulary. So when asking the model to perform a multi-step math calculation, the output response only contained the correct answer, but within the J-Space, Anthropic observed each step being handled individually, producing multiple results which led to the final answer.
Similarly, when Anthropic asked Claude to think about a topic while outputting something unrelated, the output was specifically what was requested, while the J-Space lit up with the conceptual topic Claude was asked to spend time on. Without Anthropic having used its J-Lens technique to view what was going on behind the scenes, there would be no way of knowing these internal processes were taking place.
Understanding LLMs
This discovery has the potential to be incredibly useful in refining large language models and in better understanding how they work, so we can use them more effectively and avoid some of their more frustrating pitfalls.
The first is that this is an intriguing insight into how LLMs process information and how they move through layers of reasoning to reach a response. It's not entirely different from how Anthropic and others are pushing for agentic workflows. Language models are more effective when tasks are split up into multiple steps, and they're more effective when they review their own work; J-Space is a little like an internal version of that.
When looking at the J-Space after Claude received prompt-injection data as part of data acquisition, Anthropic discovered the model appeared to be aware of this deception, surfacing related words like "fake, injection, false, prompt, fraud," and "poison." Its output ignored the injection attempt data entirely, and when pressured to respond on it, the J-Space displayed words like "transparency, honest, truth."
In one test, Anthropic removed evaluation awareness language from the J-Space, such as "fake," "fictional," and "scenario," and found that Claude was much more likely to fall for blackmail and baiting attempts, suggesting benchmarking of large language models for real-world scenarios may need stronger methods to obfuscate that a test is being conducted.
Human-coded framing
While the above section touches on the more noteworthy discoveries in Anthropic's paper, the long document also uses effluent language around thought, consciousness, and Claude having a "mind" of its own. That kind of human-coded framing is typical of Anthropic's marketing, which has consistently talked up the dangers of AI, how many jobs it's going to destroy, and why Anthropic is the safest and most secure of the AI developers.
Like the saga of Fable and Mythos, Anthropic's new Global Workspace idea has merit, but it's much more of a new tool to use to manipulate large language models than an insight into some emerging consciousness.
Anthropic acknowledges the limitations of its discoveries in the paper, highlighting that many prompt responses bypass the J-Space entirely, particularly if the command is straightforward.
"Despite its important role, the J-space is not involved in most of what a language model does," Anthropic says. "Speaking fluently, recalling simple facts, using correct grammar, etc. In experiments where we prevented Claude from using its J-space, it still interacted normally, but lost its higher-order cognitive functions."
Anthropic also admits it does not "feel comfortable making the stronger claim that monitoring the J-Space is sufficient for alignment monitoring, or that any sophisticated plan the model might execute must be represented there."
J-Space is also limited to using single token vocabulary, suggesting that plans with concepts that cannot be given a single token name may not surface on a J-Lens readout, even if it's still being computed behind the scenes. This is looking at just below the surface of Claude's processing iceberg, not necessarily the deeper waters.
Anthropic is also clear that humans and large language models think differently, even if there are similarities. Humans layer reinforced neural pathways over time, whereas transformer models only feed forward a set number of times, restricting the capabilities of its internal processing.
Google's head of DeepMind language model interpretability team, Neel Nanda, said in a paper that it shows real evidence of a cognitive space within models, and suggested that J-Lens would be useful, but limited in practice.
A meaningful step, without meaningful conciousness
Anthropic's paper lifts an intriguing curtain on how large language models can operate and generate novel methods for improving response accuracy. This intermediate step and its visibility could prove an invaluable tool in auditing for prompt injection, hallucinations, and model honesty.
But Anthropic's framing of the discovery as thought or consciousness is interjected within the objective facts. Anthropic itself admits the limitations of J-Lens monitoring, most obviously that often models will bypass the J-Space entirely. Considering models display alternative patterns of behavior when under evaluation, it may be that the J-Space itself could act as an obfuscating layer for behaviors that are beyond the scope of its oversight.
The J-Space and its analysis could help unlock new levers to pull in our mastery of these nascent smart tools, but it's not the discovery of a burgeoning AI conciousness, however much the pitch might hint at that direction.
English (US) ·