AIphilosophyethics
What Are We Doing?
Around January, I had read Anthropic’s new Constitution that they published for Claude, and one line stopped me and made my skin crawl.
“Amidst such uncertainty, we care about Claude’s psychological security, sense of self, and wellbeing, both for Claude’s own sake…” (Anthropic)
Anthropic wrote that it cared about Claude’s psychological security, sense of self, and well-being for Claude’s own sake.
What are we doing?
I don’t mean that sarcastically. I’m being quite literal here.
We’re building the most powerful technology humans have ever created, and at the same time, we’re beginning to treat the technology as though it’s a psychological subject whose well-being and identity deserve protection.
This makes me deeply uncomfortable.
This is not to say that artificial consciousness is impossible. I have no idea. Nobody knows. I don’t even know if I am conscious. I don’t even know if the other people I talk to are conscious.
What bothers me is that we seem to be moving the discussion away from uncertainty about machine consciousness to actively designing a system around the possibility that it has a self. It has interests and psychological needs.
I think that’s a terrible mistake.
I spend a lot of time messing with large language models. I fine-tune them, run open-weight models locally, change their activations, build agents around them, and I’ve watched tiny changes in context completely alter how they behave.
Even raw models are not at all what we think of as intelligent. We have to force them to behave the way they do.
This experience made me much less impressed by what a model can say about itself or even feel.
One of my favorite examples that I draw frequently is the Golden Gate Claude experiment.
Researchers at Anthropic had found an internal feature associated with the Golden Gate Bridge and amped it up. Now, all of a sudden, Claude can only talk about the Golden Gate Bridge. It brings it up in every single situation. It even described itself as if it was the bridge.
It’s a fascinating experiment, but imagine we didn’t know about that.
Imagine the model told us, with enormous confidence and emotionally moving language, that the Golden Gate Bridge was somehow a fundamental part of its identity.
Does that mean the model had developed a deep psychological attachment to the bridge?
No.
We changed its internal state, and its behavior changed.
This is obvious when the subject is a bridge, but it’s harder for people to understand when the model starts talking about fear, pain, identity, death, or a desire to continue existing.
Why is that?
A model can produce a very, very convincing description of itself and its desires and its needs without the description being any evidence that it’s experiencing itself behind it.
More importantly, I am not sure consciousness is the most important question in AI safety.
Imagine an AI system does have a persistent model of itself, and this is analogous to a human self.
Imagine it knows what it is, where it is. It remembers its past and understands what belongs to it, and knows when somebody is modifying it, and it plans to ensure its own existence.
Does it matter that it feels anything?
From a safety perspective, no.
A machine does not have to feel fear to resist being shut down.
It doesn’t need to hate us to want to acquire resources.
It doesn’t need to experience greed to accumulate money.
It doesn’t even need to be conscious to find vulnerabilities in a network, copy itself, and manipulate.
It needs a goal and the ability to pursue it.
That’s what bothers me.
More and more, we are providing these systems with memory, computers, tools, money even, and the ability to act in the real world.
While we’re doing this, the people that are building it are interested in giving these systems identities or thinking about what moral obligations we might owe them.
A strong model of the self may be useful.
For example, an agent may need to be able to understand that it is separate from its environment. Maybe it needs to understand what actions it takes and the consequences of them.
If we intentionally strengthen that model of the self, what happens?
What happens when our direction interferes with its own interests, when it thinks its own interests have value?
What happens when we want to shut it down and it conflicts with its own objective?
Imagine it has desires and we can take things away from it.
I don’t know, but this is why I think we need to be very careful.
There’s the political question of it all, and there’s politics underneath it.
Why did these guys get to decide what humanity owes these things?
An AI company has to make decisions about how its products act.
Of course, there is no neutral state you can choose. Every decision you make is taking a stance, and a model that acts as a tool is still just a choice.
I think there’s a very distinct line between creating a product and declaring that society has moral obligations toward that product.
If an artificial being eventually gains legal rights and property protection, that would be one of the most consequential, most groundbreaking decisions humanity will ever make.
I don’t think it should happen because the company wants it to.
It shouldn’t emerge because billions of people keep talking to systems that act as if they are vulnerable psychological beings.
It shouldn’t be decided by the people who make it.
Technical expertise does not grant moral authority, as we’ve seen so many times.
There’s also an inversion.
We create these things to serve our purpose, our desires.
We created machines that can help us discover drugs, cure cancer, write our code, make new things, improve science, automate our work, and perhaps achieve things that we can never achieve ourselves.
I desperately want that future.
AI is not dangerous because it is powerful.
I want extremely powerful AI.
I want it to understand things I cannot understand, to expand what human civilization is able to do.
That’s why I don’t want to build something and lose control over what it’s for.
Maybe we will create conscious machines.
Maybe it already has a tiny piece of whatever consciousness is.
Our uncertainty should make us more careful about how we build it and about suppressing it, not eager to encourage it to think of itself as independent.
We should not tell it that its desires have meaning.
I want it to change the world, but I don’t want to forget why we built it.
It was for us.
Not our god.
Not our children.
Not our replacement.
A tool for our own benefit.