The Memory That Says "I Don't Know"
A faithful memory layer for any agent. One that cites its sources, refuses to make things up, and is built to outlive me. The third thing I have shipped this way.
An elephant never forgets. It is the oldest line we have about memory, and I have spent a while now building the thing it points at: a faithful memory layer. Not for my agent. For any agent.
Somewhere in the building I coined a name for what I am really after. Tusks of truth. The part of the record that is hard, and permanent, and yours. The part that does not get quietly filed away because someone would prefer you forgot. The truth will always prevail, but only if something keeps it while it waits. That something is what this post is about.
A few months ago I wrote about a wiki I built to stop leaking knowledge, and then about a calendar that blocks time using nothing but email. Same shape both times. Find the pointless friction, hand the relentless part to a machine that does not get bored, keep the judgment for myself. This is the third thing I built that way, and it is the one the other two were quietly leaning on the whole time.
It is memory. A single faithful record that everything else reads from.
The wiki worked, and that was the problem
The wiki did exactly what I wanted. It caught the knowledge before it leaked. But once it was actually good, something happened that I did not plan for. I started to trust it. I stopped re-deriving things, because the note was there and I believed the note. That is the entire point of the exercise. It is also the moment the thing becomes dangerous.
A memory earns a new failure mode the instant you rely on it. It can be confidently, quietly wrong, and you will not catch it, because the whole reason it exists is so that you do not have to look. On a low day, and energy is still a budget I am rationing, I am not going to check. I am going to take the first plausible thing and move.
The wiki survived this because of one rule I built in early. Verify before write, not after. Nothing entered without a source, without passing a few checks, without flagging its own contradictions. I called it the Guardian. It was the most important part, and it was the seed of this whole next thing.
The convenient kind of memory
There is a popular way to give an agent a memory, and it is seductive. You let the model tidy it. The model reads your history, writes a smooth summary of who you are, updates that impression as it goes, and hands the impression back to itself next time. It feels like magic. It is also a game of telephone with a single player. Every pass is a small rewrite. The summary drifts from the transcript. The profile slowly becomes a story the model tells about you. And a tidy story is easier to recall than a pile of facts, so the story wins.
For plenty of uses, that is fine. For a memory I actually lean on when I am tired and not double-checking, it is precisely wrong. I do not want a memory that is confident. I want one that is correct, and honest about the difference.
Three rules, all of them refusals
So I built one with three rules. What is striking, looking back, is that all three are things it is not allowed to do.
It stores what happened, not its impression of what happened. The record is the record. Nothing rewrites it in the background.
It is allowed to say “I have no record of that.” Out loud, instead of guessing.
Everything it tells me carries where it came from. Every answer is a citation, not an assertion.
None of these are clever. All of them are discipline.
Why “I do not know” is the hard part
That middle rule sounds trivial. It is the whole game.
A model is trained to be helpful, and helpful, under pressure, means filling the silence. Ask most systems something they have no answer to and they will hand you a thing shaped like an answer, because returning nothing feels, to them, like failing. For a memory layer, the thing shaped like an answer is the failure. The single worst output it can produce is a confident yes to a question whose true answer is “there is no record.”
A small example. On a flat day I asked it when I had last spoken to someone about a particular thing. The wrong kind of memory gives you a date. It feels right, and you believe it, because that is what flat days are. What I got back was two lines: the exact exchange, and where it came from. And on the day it had nothing, it told me it had nothing. That was the answer I needed too. A false “last Tuesday” is not a smaller mistake than a missing one. It is a larger one, wearing a smaller coat.
So the hard part was never retrieval. It was teaching the thing to look properly, and then to accept the silence when the silence is real, and to say so plainly.
How it finds things: the shape, not the recipe
I will stay at the level of shape here, because the details are the part I am keeping.
Recall is not one search. The tempting design, the one almost everyone reaches for first, is to turn everything into vectors and ask for the nearest match by meaning. That works until it does not. Similarity search is fluent and vague. It is very good at finding something related, and very bad at telling you it found nothing, which is the exact behavior I was trying to design out.
So the layer does not trust any single lane. It runs several at once. An exact lookup for the thing you literally named. A full-text search over the real words. A meaning-based search for the ones you did not use. Then it fuses the separate rankings into one, so a fact that shows up strongly across different kinds of search rises to the top, and a lucky single hit does not run away with the answer. The fusion itself is a known technique, reciprocal rank fusion. The interesting part is which lanes, and how they are weighed against each other, and that part stays in the box.
Underneath all of it is a floor that never touches the model at all. Dates, times, counts, how long ago, what came first: arithmetic, done in code. The model sits downstream of the facts. It never gets to invent one. When the answer is a timestamp, a timestamp is computed. It is not recalled, and it is certainly not guessed.
Then I handed it to T9i
Once memory is a layer instead of a pile of files, the interesting question is the same one the calendar arrived at. Who is doing the reading and the writing.
My agent, T9i, does. But the layer does not belong to it. T9i is simply the first to read from it, and it was always meant that way, for any agent. It does not keep a private notion of me in its head between sessions. It queries the layer, the layer answers with citations, and when there is nothing, it hears “no record” like anyone else would. The knowledge and the identity live in the layer, not in the model and not in the app sitting in front of it. Which means they are portable. I can change the interface, or the model underneath, and the memory comes with me. So does the person it knows. The agent is a reader of a faithful record, not the author of a convenient one.
It is the same partnership as before. I hold the shape and the judgment. It holds the details and never gets bored of them. The difference this time is that I built the details a spine, so that neither of us has to take them on faith.
The receipts
I did not start this in the abstract. I have nine years of notes. Hand-typed, dated, kept. Not my memory of what happened, the actual record of it, set down close to when it did. They are all indexed now, and the brain holds them, so any one of them is a question away. If someone asks me what was said on a particular day, years back, I do not reach for my recollection. I reach for the record, and the record answers, with the source attached.
That is what I mean by faithful. Not a feeling. A citation with a date on it. I have the receipts, and the entire point of this layer is that they stay exactly as they were, findable, for as long as I need them, which is the part I care about most.
Because it has to outlive me. A memory that dies when I do was never really a memory. It was a convenience I rented. The record I am building is meant to stand on its own, after the app that shows it, after the model that reads it, after me. If a thing is true, it should not need me alive to keep being true.
What this is
Let me be clear about what this is not. It is not the model remembering you. That is the thing nearly everyone is shipping, and it is a different, softer promise.
This is a substrate with one job: keep the record straight, so that the useful, fallible thing on top of it can be useful without being trusted blindly. The human stays in control. The agent does the work. Nothing is asserted without a source, and nothing is answered that has no answer.
The wiki taught me that a memory you cannot trust is worse than no memory at all, because it manufactures false confidence. This is that lesson taken all the way down. It is not open yet. This one I am sitting with a while longer. But it is live, it is what the other two have quietly been standing on, and it is the part of the robot I trust the most. Precisely because it is the part most willing to tell me it does not know.
The truth will always prevail. It has always just needed somewhere faithful to wait. That is what this is, and it is built to wait a very long time.
That is three things now, shipped the same way. There will be more.
Move forward.