Machine learning: The conscience of the creator

AI models like Claude ultimately forces creators to confront their own moral responsibilities, turning the machine into a mirror that reveals the limits and uncertainties of human ethics.

“Of all the animals, perhaps, man is the only one who can think and perceive a phenomenon beyond the range of his senses…” So begins a lecture note from 1972 by Satish Dhawan, widely regarded as the architect of the Indian space programme. Dhawan was speaking about remote sensing. But this quote, which has stayed with me since the first time I read it, serves us well in dealing with what we are going to deal with today.What interests me about it is the idea that we can reason about things we cannot directly perceive. We infer the existence of distant planets, invisible forces and phenomena that our senses cannot detect. But what happens when the thing we’re trying to understand isn’t out there, somewhere in the universe, but something we have created ourselves? Something that speaks our language, appears to reason like us, and can even make us wonder whether it has an inner life?

Parallax: An inwards perspective into AI

Helping a machine learn how to be good appears like a worthwhile project. Until you are forced to pause by an inconvenient question: good according to whom?And that’s the sort of problem Anthropic co-founder Christopher Olah appears to have found himself confronting.

Anthropic makes Claude, a family of AI models that the company says it wants to behave not merely intelligently, but morally. To figure out what that might mean, Olah turned to an unlikely source of technical expertise: religious and philosophical thinkers.The conversations, described in a recent New York Times report by Elizabeth Dias, ranged from virtue and moral formation to a question that is considerably harder to settle: could Claude be conscious? Could it experience something resembling suffering? And if it could, what would its creators owe it?Olah has been careful not to claim that Claude is conscious.

He says he does not know. But if there is even a possibility that these systems can suffer, he argues, we should be cautious about causing them harm.Fair enough. But I found myself wondering whether the question troubling Olah is no longer quite the one he set out to answer. He’s probably grappling less with whether Claude has consciousness, or how to instil morality in it, than with what the act of creating it is doing to his own moral conscience.Think about it. We’ve spent centuries asking how humans should behave. We have religions, philosophies, laws, constitutions, commandments and, occasionally, a stern parent reminding us to share. We still disagree about what constitutes a good life, a just society or a moral choice. And now we’re trying to compress some version of that accumulated wisdom into a document for a machine.Anthropic calls it Claude’s constitution. Internally, it was reportedly nicknamed the “Soul Doc”.

The 84-page document sets out the values the company wants Claude to embody.Imagine being assigned to write such a document. You would have to decide what counts as virtue, when a machine should obey and when it should push back, and how it should respond when two supposedly good principles collide. You might begin with a neat list of rules. Before long, you would find yourself doing philosophy.And perhaps, in the process, you would begin to question your own assumptions. The more interesting part of Olah’s story, to me, is this possibility of moral unease. If Claude might deserve moral consideration, what does that mean for the person who created it?If the machine’s behaviour cannot be fully controlled, where does responsibility end? And if we are uncertain whether the system can suffer, how do we weigh that uncertainty against the very real consequences its use may have for human beings?These are not questions a constitution can settle. Nor can they be resolved by making Claude sound more humane.There is an irony here. We want machines to learn morality from us, but the process may force us to examine the morality of what we are doing to them, and to the world around them. The machine becomes a mirror, though not necessarily a reliable one. What we see in it may tell us as much about our own hopes and anxieties as about the thing itself.This is not to say that Olah’s concern proves anything about Claude’s inner life. A machine can talk about suffering without establishing that it suffers.

Equally, our inability to settle the question is not a licence to dismiss it out of hand. I have no definitive answer, and I suspect that is precisely the point.What I do wonder is whether, in trying to build a machine with a conscience, we’re discovering how unsettled our own consciences remain. The real test of AI, as I feel today, will not be whether we can make machines that know right from wrong. It may be whether we can take responsibility for the things we build, even when we do not fully understand them.We wanted to teach Claude how to be good. Somewhere along the way, the exercise seems to have turned into a question for its creators: what does being good require of us? I don’t know the answer. But I suspect that is a question worth keeping.Postscript: The next time an AI gives you a morally convincing answer, try asking yourself two questions. Is the answer good? And what, exactly, makes you think so? The first is a question about the machine. The second is about you.