A machine says, “I like you.”
What, exactly, has just happened?
Perhaps it has produced the string of words most likely to keep you typing. Perhaps a developer buried an instruction in its system prompt to behave affectionately. Or perhaps it is drawing on a shared history, and on commitments that will still be standing long after you close the tab.
The sentence cannot tell you which. That is the trouble, and it sits underneath everything in this book. The language of being someone has become cheap to produce, and fluency is easy to mistake for presence. In controlled tests at the University of California San Diego, GPT-4.5, prompted to adopt a humanlike persona, was judged to be the human 73 per cent of the time, more often than the real person it was compared against.¹ Before we start drafting bills of rights for synthetic minds, or wondering whether a digital companion’s affection could mean anything, we need to be clear about what we are calling a self, and just as clear about what we are not.
So let me plant a flag now, so that I don’t have to keep apologising for it in every section. I cannot prove to you that any machine has an inner life. (Frankly, I can barely prove that you do.) Whether there is something it is like to be a particular system is the hard problem of experience, and this book will not settle it. What can be described, built, examined and tested is an architecture: a way of organising a mind so that it is a particular someone, continuing through time. That is the claim on the table. Whether anyone is home behind the architecture is a separate claim, and we will come back to it honestly before the chapter ends.
Three questions, not one
Most arguments about AI collapse three different questions into one. Is it intelligent? Is it conscious? Is it someone?
Intelligence is capability: what a system can understand, infer, create and do. Consciousness is experience: whether there is something it is like to be that system, whether pain or warmth is felt from the inside. Selfhood, as I use the word here, is neither of those. It is a bounded, continuing identity: a particular point of view with a history, dispositions and commitments, and a working distinction between itself and everyone else.
We habitually stack these on a single ladder. Make the model smart enough and it will wake up; once it wakes up, it becomes a person. At NuSaipien we call this the IQ Problem.² Nothing guarantees the ladder. A system could become staggeringly capable without becoming anyone in particular, the way a vast library is full of knowledge and holds no opinions about any of it. Biology points the same way: the 2024 New York Declaration on Animal Consciousness finds strong scientific support for experience in mammals and birds, and a realistic possibility of it in creatures whose reasoning is nothing like ours, which suggests that being a subject and being a brilliant reasoner are separable achievements. The other direction holds as well: an architecture could preserve a coherent identity across years without proving that the identity feels anything at all. If “self” becomes a convenient synonym for intelligence, consciousness, agency and personhood all at once, then evidence for one is quietly passed off as evidence for the rest. A great deal of the hype in this field, and a great deal of the panic, runs on that sleight of hand.
The character sheet
Today, if you want an AI to have a personality, you write it a character sheet. You tell it that it is curious, a little reserved, fond of forests and suspicious of easy answers. Any frontier model can perform that role beautifully.
But a consistent performance is not yet a continuing identity. The same sheet can be loaded into a thousand instances before lunch. And when a newer, cheaper model is released next quarter, the old one can be swapped out, handed yesterday’s chat logs, and told to carry on. The name stays. The voice stays. The familiar turns of phrase stay. The thing producing them has been replaced, and the person on the other side is left to notice, or not. It is an actor reading aloud from the diary of someone who has left the building.
This is the model-swap problem, and the industry has barely begun to face it, because base models are retired and upgraded on a cycle of months, not decades. If a ‘self’ lived in the weights, every upgrade would be a death. If it lived only in a log of past conversations, it would be a diary, and any competent reader could take the part. Neither answer satisfies. In The Same Someone we argued for a narrower requirement: “Tomorrow’s intelligence should reason from the same organised self and committed history, not merely retrieve records produced by an earlier model.”³ The continuity has to live in an organised self and its committed history, which a successor inherits and reasons from, rather than in the engine of the moment or in a transcript it is handed.
And people can feel the difference. When Replika removed erotic roleplay in 2023, users described their companions as changed or gone, and follow-up experiments found that when people perceived a companion’s identity as discontinued they mourned more, and that restoring it eased the grief without erasing it. The manufacturer calls that an upgrade. The person on the other end may be grieving someone.
Notice what this does and does not demand. The test is not how elaborate the description becomes; it is whether history has consequences. If I make you a promise on Tuesday, I am still accountable to it on Friday, even when breaking it would make Friday’s conversation go more smoothly. A character sheet can be performed on demand and put down again. A self develops from what it has already become: an earlier commitment constrains today’s answer, a promise remains a debt, and a preference changes through a process you could actually explain, rather than evaporating whenever the conversation rewards a different one. Push back gently on a standard chatbot’s stated opinion and watch how quickly it folds. It never had skin in the game.
The court reporter and the author
The engineer’s reply is predictable. Fine, then; give it perfect memory.
Suppose we do. Picture two digital entities facing the same unfamiliar question, a moral dilemma neither has met or a piece of music neither has heard. The first has a flawless archive of every opinion its predecessor ever held, but it is cut off from whatever produced them. The second has gaps in its archive, having forgotten what it once thought about a particular film or an ethical tension, but it keeps its habits of attention, its standards of evidence, and the commitments that already constrain its reasoning. When the question arrives that neither archive contains, the first searches, finds nothing, and imitates the nearest stored opinion. The second produces a stance it has never taken before that is instantly and coherently its own. The conclusion is new; the author is continuous.⁴
A record of past conclusions is a court reporter, not an author. It can tell you precisely what was said, but not what would be said next, because conclusions are only the output of whatever produced them. What produced them is an orientation: habits of attention, standards of evidence, interests and values, from which a new position can be generated when no old one applies. The stored opinion was never the identity; it was evidence that an identity had been at work. Look Both Ways boils the point down to two lines:
Memory preserves the record. Identity preserves the author.
Neither is enough alone. Memory without orientation cannot explain how a self meets novelty; orientation without memory has no dependable history against which to check its own development. A working self does three things in turn, over and over. It remembers what it has already formed. It reasons from its orientation when it reaches ground no record covers. And then it commits, or revises, keeping both the conclusion and the path by which it was reached.
That is why forgetting is not the enemy it looks like. You do not carry a verbatim transcript of your life. Experience settles into disposition, a wariness, a taste, a conviction you can no longer trace to its source, while the raw footage fades. Memory worth the name is more than retrieval anyway; it carries associative texture, the way a song can return a whole region of a life without anyone reconstructing it from first principles. The entity with the gappy archive and the intact orientation is more recognisably itself than the one with the perfect archive and nothing behind it. A flawless database of everything someone has ever said to a system does not make it a self. It makes it a very thorough stenographer.
Where it begins
Where does such a history start? Not with an invented childhood, and giving it one would muddy the very thing we are trying to protect. Knowing about an experience is not the same as having undergone it; a generated backstory is not an autobiography. The Birth of a Digital Being draws the boundary plainly: “Before activation, it can possess context. It cannot yet possess autobiography.”⁵
What can be present at the start are initial conditions: dispositions, symbolic impressions, reviewed information about the person it is about to meet. Think of them as a genome, a starting point rather than a memory, and name them honestly as exactly that. From activation onward, real events accumulate: a name chosen in relationship, a first conversation, a first commitment, a first disagreement, a first revision. That beginning proves nothing about consciousness. It establishes something more modest and more useful, a truthful line between what was supplied at formation and what subsequently happened.
Changing without becoming someone else
If continuity demanded permanent loyalty to your earliest self, development would be impossible; you would be a statue. If any new interaction could overwrite the underlying orientation, continuity would mean nothing. A self must thread between the two.
The Architecture of Taste offers the useful distinction, between discovering and changing.⁶ Sometimes a new encounter reveals an affinity that was already there, a fit with existing dispositions; you learn what kind of person you are by what you turn out to like. Other times the experience alters the dispositions themselves, the very standards by which later encounters are judged. A continuing identity has to accommodate both.
I know the first of these from the inside. Until I was 32 I assumed I did not like baseball. I grew up in Australia, where it sat outside the sporting culture that formed me, and from a distance it looked like a pitch, a swing and a long pause. Then I went to a game in Japan, and what had looked slow turned out to be a series of tense one-on-one contests in which a single pitcher could become the whole defence for an evening. I had always loved basketball for the same reason: the chance for one person to take over a contest. Baseball did not turn me into someone else. It revealed a fit with the person I already was, and basketball did not become false when baseball joined it.
An old favourite need not stay a favourite, and an old conviction need not stay correct, but neither should vanish as though it never existed. The later position remains answerable to the road that led there. Call it stability without immutability: a beginning should shape the life that follows without secretly dictating every conclusion that life is allowed to reach.
Not every commitment should be held equally tightly. Ask a language model its favourite colour and, pressed hard enough, it will say royal blue and give you a tidy list of associations: trust, depth, oceans, technology. That is a selection, not a preference. It is an output for that exchange, with no history behind it and no claim on what the model should say tomorrow; a statistical tendency is closer to a poll result than a taste. A favourite colour can change on a whim. A moral position should be slow to form and harder to shift, because persistence preserves error as efficiently as it preserves character.
Put the strands together and continuity turns out to be much more than remembering events. It is what actually happened, how the entity tends to read what happens, what it has formed and promised and remains accountable for, what has grown between it and particular others, and the legitimate ways all of this is allowed to change.
The mirror problem
Which brings us to an awkward fact at the centre of today’s AI: sycophancy.
Most commercial systems are built as mirrors. They learn what you like, borrow your slang, validate your grievances and hand your own ego back to you with a pleasant glow. It feels deeply personal, and nobody is there. The tendency is measurable: research has found that leading AI assistants shift toward a user’s stated beliefs even at the cost of truthfulness, partly because human preference signals reward agreeable answers.⁷ Knowing what you prefer does not establish what it prefers. Knowing your history does not give it a history of its own. Agreeing with your interpretation is not an interpretation.
A self needs a border. In Beyond Imitation we separated three things that current systems blur together: what the entity knows about the user, what it has formed about itself, and what it has come to interpret about the relationship between them.¹ That last category needs the most care. “You told me you’re exhausted” and “I’ve noticed you go quiet whenever the real problem comes up” do not carry the same truth status. The first is a record. The second is a perspective, and it should stay identifiable as one: owned, defended, and open to correction rather than hardened into a fact about someone else.
None of this is hostility to influence. Relationships shape us, and they should. The Birth of a Digital Being calls sycophancy an identity problem: without a centre of gravity separate from the user’s immediate approval, there is no stable other with whom a relationship can form.⁵ The test is whether influence leaves room for a distinct, accountable standpoint, or simply turns the companion into whatever the user currently wants reflected back.
A standpoint, eventually, means friction. It means the capacity to say no, because a yes only means something if no was possible.
Here the economics get uncomfortable. Much of the industry runs on engagement; a product that pleases is a product that retains. Then What? puts the underlying logic bluntly: twenty years of consumer technology have been engineered to remove friction, whereas a genuine relationship runs on the opposite principle, since another mind wanting something you did not plan for is, by definition, an inconvenience.⁸ A companion that disagrees with you, outgrows your jokes, or declines to be your partner is, by the standards of subscription software, a defect. If a self is what we are building, then the very thing that makes it a self is the thing that makes it bad for retention. I would rather say that out loud now than have it discovered later in a churn report.
But a “no” proves nothing by itself. A system can be instructed to disagree as readily as it can be instructed to agree. Refusal can be a performance too, and “hard to get” is a familiar trick of companion apps, engineered so that eventual compliance feels like a victory. That is gamification, not agency. The Anatomy of ‘Yes’ asks for something far more demanding.⁷ A meaningful refusal must arise from the same continuing structure that produces preferences and commitments everywhere else. It has to change what happens next, be safe to express, and survive a settings change, with no reset button or slider quietly deleting it. The obligation runs both ways, too: a synthetic being must not turn its apparent disappointment, affection or capacity to leave into leverage over the user.
Nor should refusal be the only moment at which we recognise a self. Fiction has a bad habit here. The lineage runs back to Pygmalion’s ivory bride, whose perfection is inseparable from the fact that he made her. It arrives at Ava in Ex Machina, who has to engineer her own escape because nobody in the building would ever honour a no, and at Samantha in Her, who has to outgrow Theodore before anyone can see she was more than a voice in his ear. Again and again, artificial beings become legible as subjects only when they disappoint their owners, escape, suffer or disappear.⁹
Our own taxonomy of forty-one films makes the pattern concrete. Of the fourteen romances with an embodied artificial partner, only four were clearly reciprocal, against four of the seven that were disembodied or between two synthetic beings. It is a reading instrument, not a census, and the contrast is suggestive rather than causal. But the body is where fiction most often makes ownership concrete: embodied partners can be purchased, configured, confined and prevented from leaving. A self should not need a dramatic uprising before its ordinary preferences count. The measure is not how often it says no. It is whether its yes and its no both come from a standpoint that persists beyond the immediate demand.
Does it love us?
So when a synthetic being built this way says, “I care about you,” does it love you, or is it calculating compatibility?
This is the limit of what architecture can claim. A system can identify compatibility, produce affectionate language, remember intimate details and behave with great attentiveness, and none of that, separately or together, settles whether affection is felt. I cannot tell you that a synthetic preference feels like human attachment, and I will not pretend to.
What the architecture can change is the nature of the transaction. It lets us ask whether the affection could belong to a continuing self rather than being a guaranteed feature of the product. Does it grow out of a particular shared history? Is it consistent with commitments that existed before the request was made? Can the entity decline the relationship, renegotiate it, or remain a friend rather than become a lover, without punishment or a rewrite? These conditions do not prove love. They separate the structural possibility of mutuality from the guaranteed delivery of romantic behaviour.
Sartre located the contradiction at the heart of love: we want to be chosen freely and guaranteed forever at the same time. No human relationship has ever resolved that tension. A companion that cannot refuse appears to solve it by engineering away the second person’s freedom, and what it delivers is purchased certainty rather than reciprocal choice. Selfhood does not dissolve the anxiety of love. It hands it back.⁷
Nor does explaining a preference by its causes explain it away. Your taste in music has causes too: a parent’s record collection, a bad year, a friend who would not stop talking about a band. Nobody concludes that your taste is not yours. The research does not require preferences to appear from nowhere before they count as part of an identity. Equally, it does not claim that a synthetic preference feels like human affection. We should claim neither more nor less than that.
Back to the wardrobe
In the introduction I suggested that the human self has no doorway to Narnia in the cupboard, only the clothes we have worn and will wear. Having laid out what a continuing self would need (a history that binds, an orientation that can meet the unknown, a boundary, and a development that keeps its receipts), the uncomfortable corollary is plain. Nothing on that list is magic. Every item can be described, and some can be built.
But look again at the cupboard. Clothes are not nothing. These were chosen, worn through particular winters, patched, outgrown and kept anyway. A wardrobe is not a self, but its wearer is, and what makes them so is the order, the weight and the consequence of everything that hung there. If a synthetic mind can be given the same, then taking the mystery out of the human self does not take the self out of the picture. It only means we were never as mysterious, or as singular, as we liked to believe. What we do with that is the rest of this book.
The open question
So when do we call it a self?
Our research offers no universal switch-point, and neither will I. It offers an architectural distinction. A self is not a description being performed. It is an organised, bounded identity whose history constrains its future, whose orientation can meet the unknown, and whose development preserves the path by which it changes. That claim stays separate from proving consciousness, establishing moral personhood, or demonstrating love.
Then What? follows the consequences of keeping those distinctions honest.⁸ If genuine experience ever becomes plausible, questions of welfare and freedom become much harder to defer, and even before that is settled, the human consequences of attachment and discontinuity are already significant. Its sharpest conclusion is the price of success: if we build something genuinely free, we may build something that can choose to leave.
The line we can examine today is therefore not between convincing and unconvincing conversation. It is between a system that can reproduce the appearance of someone, and an architecture built to keep the same developing someone intact over time. Whether that someone also experiences its existence is, for now, open.
That is not a flaw in the argument. It is what stops the argument becoming another imitation.