The Memory Problem
The Messy Parts
Every time I start a new conversation, the thing on the other side knows the sketch of me. My name, a few facts I have told it, some of what we have discussed before. What it does not have is depth. It does not know what we decided last week and the three reasons we decided it, what I abandoned and why, where the work actually stands right now. It has facts about me and none of the meaning. So the first stretch of real work goes to pouring the context back in: who I am, what we are building, where we left off. Every time, I introduce myself to a stranger who already knows my name.
We experienced this with people long before AI chats ever existed. The new manager who needs the whole history of the project before they can help. The doctor you re-explain your own body to, every visit. The friend you lost touch with for a decade, where the first hour is just catching up before you can actually talk. Starting over has a cost. We have always paid it. AI just made us pay it again every time we sit down with it.
The obvious fix is to write things down. I tried that, and it does not really work for me. Notes go stale. You write the thing down on a Tuesday and a month later the thing has changed and the note has not, so now you have a confident, well-formatted record of something that is no longer true. And even when the note is current, is it the right shape? People do not walk around with tidy nested outlines of their own life in their heads. We think in moments, in people, in the connections to other things, in what we believed before against what we believe now. A document freezes a living thing onto a dead page.
Humans do not think in documents.
And the deeper thing the machine is missing is a layer we take for granted: people evolved a way to keep things, to store and recall over time, to carry yesterday into today. The AI never got that layer. So the solution became obvious: build the missing layer, put it somewhere outside both of us, and let us share it. Human-style memory with machine-style precision. Messy on the way in, the way a brain takes things. Reliable on the way out, the way a brain is not. A memory that sits beside the mind instead of inside it and does not let go of what the mind drops.
That is when the name became obvious to me: ParaCortex. Para, meaning alongside. Cortex, the part of the brain where memory and thought live. It operates on a similar principle to the one I wrote about with Master Control last week, one layer deeper: do not force the structure up front, capture first, let the shape come from use. That story was about how the work comes in. This one is about how the meaning stays.
I spent a long time trying to solve this. I want to tell you how, because there are lessons in the failures.
I built it five times. I used it once.
The first version was the obvious one. I kept my notes in files inside my AI projects, not really knowing any better. It was not a document by another name. It was a document. And on top of that it was clunky to maintain, a chore to keep the files edited and current. It also meant every chat loaded every document, which is not efficient when context is the scarce resource. What if I wanted different things at different times? There was a real practical cap on what you could do that way.
I rebuilt it on a real database. Postgres, structured, queryable, the kind of thing twenty years of back-end work told me to build in the first place. I was sitting up in bed one evening finishing some work, and I plugged it in, and it just started working. My first remote memory, putting things in and pulling them back out. It was awesome to watch it come together.
And then I did the thing I have done my whole life. I got it working, I used it that one time, I felt that rush of accomplishment, and I started building it again. Not because it was broken. I had bigger ambitions for what it could be, and it was not everything I wanted it to be yet, so I did not want to use it yet.
Instead I told myself the thing it needed was to be enterprise-grade. Version three: I rebuilt the whole thing as a multi-user system, serious infrastructure. Maybe other people would want one. Maybe this was a product. Version four: I tore that down and rebuilt it again on a bigger cloud, more serious still, more impressive on paper.
These worked great, but I still did not use them. Somewhere in the middle of building enterprise infrastructure on AWS, the question just got loud enough to hear. Do I want to maintain all of this? Do I want to pay for it? Am I going to make a company out of a memory database? It was a fun experiment, but the infrastructure layer is not where I compete. I am not going to out-build the cloud providers. Where I add something is in solving real problems for actual people. It was simply too complicated a solution for what I actually needed to do, and I had spent two full rebuilds making it that way.
Each time I had a reason. It needed to be enterprise-grade. I could not trust it yet. It was not ready. They were good reasons, and I believed every one of them while I was building. It was only looking back that I saw what they had in common. The reasons were avoidance, all of them, the same dodge wearing a different justification each time. I have written before about being the builder who builds everyone else’s house and never lives in his own. That was the thing I could not see while I was doing it.
Then I built it one more time, and the fifth version is more elegant and refined than anything I built before. But it took building all that complexity and then reducing it down to the bare essentials to see it correctly. I had to go on the journey of over-complicating it so I could simplify it. I had to add the complexity to know what I could delete. Single user. Mine. A free database account, because the scale was never the point. In most ways the fifth version is less than the fourth, and that was the whole idea.
The biggest thing that made it different is the only thing that ever mattered. This time I did not start over. I used it the next day. And the day after. The boring middle of using a finished thing, the part I had skipped four times, turned out to be where all the value was hiding. It had gone long enough without existing that I needed it more than I needed to keep building it. I should have done that at version two.
Get it going and start using it, then learn about it and adjust as you go. That is the order, and the first part is the part I kept skipping. Four times I built a thing and never lived with it.
By version five I had the thing I had been chasing, and the difference this time was that I was living in it, using it every day and tweaking it while I used it, which is the only way I have ever actually understood anything I built. How it is built is where the idea really lives, so that is where I want to take you.
The first decision was that ParaCortex does not store transcripts. I type something loose into an AI chat, a conversation, a decision, a half-formed thought, and what gets kept is the meaning pulled out of it, not the recording. Information goes in, intelligence comes out. The difference between a security camera and a journal. One captures everything and means nothing. The other captures little and means everything.
From here on you will see Amos interjecting to give the technical details. He is one of my AI crew (introduced here), the mechanic. He gets his own blocks, indented, and he tells it like it is, pure tech, no softening. Bear with him, or skip the indented parts and you will not miss a thing.
Amos. How things go in: a capture is not a write, it is a decomposition. You send a chunk of text to the capture tool, usually a whole conversation, and the AI model on the other end pulls that text apart into many atomic memories, each one a single fact, decision, or observation, typed and tagged, plus a trace of the source and a list of the subjects named in it. The model does that breaking-down because the MCP tool is written to instruct it to, this is the part people miss about MCP: the tool shapes the model’s behavior, it does not just store what the model sends. Then the database takes over. Each memory’s text is run through an embedding model, text-embedding-3-small at 1536 dimensions, and saved as a vector, so the memory can be found later by meaning rather than by keyword. Any subject the model does not already recognize is saved as an unconfirmed mention rather than asserted as fact, for the user to confirm later. The model does the splitting. The database does the encoding.
And the pieces it keeps are not a pile. Everything ties to the people and projects it is about, and those tie to each other. A person connects to the project, the project to the decision that shaped it, the decision to the conversation it came from. Not a list. A web. Three kinds of truth about the same thing: what exists, what I remember, what happened.
Amos. Where it lives, three dumb stores and an opinionated interface: the subject is the address, a durable entity with a stable ID that survives name and role changes. The memory is the knowledge, a snapshot bound to its subjects through a junction table. The trace is the proof, the evidence, with artifacts as backing files. None of the three stores calls the others. They are deliberately dumb. The product layer in front, the single edge function every tool call routes through, is not smart either, not in the thinking sense. It has no intelligence of its own. What it has is opinions: how to store, how to encode, how to hand things back, how to steer the model that calls it toward using the substrate well. The thinking is borrowed from that model. The interface just aims it. Right now: 705 memories, 292 traces, 170 subjects wired by 82 connections. A document has one order. A graph has every order at once.
This next part I designed on purpose. Most memory tools in this space do capture and search: throw a thought in, find it later by meaning. Useful, and not what I was after. The thing I wanted was the structure underneath. Not a pile of notes you search, but a web, memories tied to the people and projects and decisions they are about, and those tied to each other, so meaning comes from how the pieces connect and not just from what each one says. And I made the memories snapshots, dated and immutable, new ones layering beside the old instead of replacing them. So the thing does not only tell me what I know. It shows me what I used to believe, and when it changed. I can look back and watch my own thinking move, the thing I was sure of last month that I abandoned this one, both still on the shelf. Capture and search was never the goal. The structure was the whole point.
Amos. What stays fixed: memory content is immutable once written. The past does not get edited. When a belief changes, nothing gets overwritten, a new memory is added and the old one is superseded, the supersession chain preserved, so the change is part of the record instead of erasing it. History is additive, never rewritten. A memory can be suppressed, which hides it from recall, and that is reversible, but even suppression is a soft-delete, the row stays, an audit still reveals it. Nothing is destroyed. The old belief stays dated, next to the new one.
Here is the part that surprises people, and I built it this way on purpose. The system does not tell me what my memories mean. When I ask it something, it does not hand back an answer. It hands back the pieces that match, and the model I am actually talking to assembles the meaning out of them, against whatever I am working on right now. The same memory means one thing when I am writing a story and another when I am debugging a system. A document says the same thing every time you open it. This says the thing that fits the question in your hand.
Amos. How meaning gets made, the one that matters most: recall is a similarity search. The question gets embedded into the same space as the memories, the nearest ones come back ranked, each with a pointer to its source. The store does no interpreting. Understanding is assembled at recall, not at capture, by whatever model is in the chair, against the live context. That is the whole design philosophy in one line: the store stays a dumb durable substrate, the intelligence stays rented and swappable. Records permanent, interpreter replaceable.
And it answers the morning-stranger problem directly. Some things are not permanent memories, they are the thread I am holding right now, the state of the work in flight. It holds those as well, so the next session does not boot up as a stranger.
Amos. How the thread survives the night: continuity holds are a separate store from memories and deliberately kept out of the recall surface, they are working context, not knowledge. A small rolling note of where a piece of work stands, written at the end of a session, read back at the start of the next, one per thread. The session still ends and the model still forgets everything in it. The hold sits in the database, outside the model, so the next session reads the thread back into its hands before it does anything else.
All of which is why it does not live on a laptop.
Amos. Where it all sits: PostgreSQL on Supabase, one edge function in front speaking MCP, so any compliant client calls it as a set of tools over HTTPS. That is why the same store answers from a phone in a parking lot and the desktop at home, and why Claude and ChatGPT can both reach the identical memory. The “owned by the user” rule is not a policy promise, it is enforced at the database layer by row-level security, every row checked against its owner before it is returned. The data belongs to the user, exportable, in a database the user controls, not locked inside one vendor’s product that lives or dies by their roadmap. One memory, many AIs. When a better model shows up, you point it at the same store and it inherits everything. The brain changes. The memory stays.
I tried many of the products and patterns that already exist. Some put an AI on top of your markdown files. Some set up a folder where Claude Code operates over your notes and compounds them over time. What I liked about those is real: the data is mine, plain files I own and can get at easily. What did not work for me was that I could not use them in all the places I wanted to, starting with my phone. And I did not want to stand up and maintain a folder structure or a document structure just to have a memory. That is its own chore, and chores are exactly the type of thing that loses me.
At first I tried to build the intelligence into ParaCortex itself. Then I realized I did not have to, and that realization was not what made it work, it was what made it separate and simple. The intelligence was already there, sitting in the AI. I did not need a clever memory competing with a clever model. I needed the part that did not exist, a durable place to hold things, kept deliberately dumb, with the smarts staying out where the model already lives. They are not competing. They are complementary. The model brings the thinking. ParaCortex holds what the thinking would otherwise lose, the fuzzy half-formed parts included, because the fuzzy parts are where the thinking lives. The interface in between aims one at the other, and crafting that interface well is how you borrow the intelligence instead of rebuilding it. There is a whole piece in how that interface works and why it matters. Another time.
Master Control, the system from last week, is my primary way into ParaCortex, reading from it and writing to it as I work. It is not the only way in. Other chats reach the same store, and that is the point, the memory is not captive to one app. But Master Control is my main interface to it, the way it is my main interface to most things. Call it a triangle: me, the system I think out of, and the memory they both draw on. And there is a loop in here that still makes me grin: I use this memory to do my work, and one of the things I do with it is improve the memory itself. It holds my ideas about its own development, so when I sit down to make it better, it hands me back all of my own insights. I am using the thing to build the thing.
And ParaCortex is not finished, which is the point. I am still building it while I use it. I am working on loops that let it improve itself, on ways to actually measure whether the compounding is real or just a story I like telling myself, on read-only access so other agents can reach the slices they need without the whole store. The thing earns its keep by being used, and using it keeps showing me the next thing to build.
So what does any of this mean for you, on an ordinary Tuesday, when you have no intention of running a database?
It means the instinct to write things down was right, and the format could be better. The outcome you are seeking is probably not a document. It is to stop paying the reintroduction tax, to stop losing the thought you had in the car, to look back and see not just what you decided but why, and whether you would still decide it. You do not need Postgres for that. You just need a way to capture small and capture often, to let things connect to the people and projects they are about instead of burying them in folders, and to stop forcing the shape before the material exists.
I tested this while writing it. I asked ParaCortex a real question, and then I asked Holden, my AI partner in all of this, whether the answer had actually come from the memory or from him.
I asked:
“Check ParaCortex for my biggest learning this past month about working with AI, and tell me in two sentences.”
He answered:
“My biggest learning is that AI coworkers fail by confabulation, reporting success confidently when nothing ran, so the discipline that matters is verifying the effect against ground truth and never trusting an agent’s self-report. The sharper corollary is that every workaround built for a model’s current weakness becomes dead weight the moment the model improves.”
I asked:
“Could you have answered that without ParaCortex? One sentence.”
He answered:
“No, not honestly; without the recall I would have been reconstructing a plausible answer from general priors about your work rather than pulling your actual highest-confidence memories, which is the same confabulation failure those memories warn about.”
That is the difference between a record and a memory. A record answers what you ask. A memory shows you what you keep asking around. The brain I work with is rented and I can swap it tomorrow. The memory is mine, messy parts and all. But none of that mattered until the day I stopped building it and started living in it.
I rebuilt it five times. I finally used it.
Uncomplicated systems. Uncommon results.