One Year of Building My Own AI Memory System

After about a year of playing with custom AI memory systems, my conclusion is both boring and surprisingly strong: the best memory layer I have built is mostly Markdown files.

Not a giant vector database. Not one huge prompt. Not a magical SaaS memory product that promises to remember my entire life. Just project-local knowledge bases, daily logs, concept notes, connections, and a few commands that let the agent maintain the structure itself.

This is deeply inspired by Andrej Karpathy's idea of an LLM knowledge base: keep the source material readable, compress it into durable notes, and make the system inspectable by both humans and models. In practice, that means my AI memory now lives in Obsidian, in plain files, close to the work.

The problem with “just remember everything”

The tempting version of AI memory is universal memory. Give the assistant every note, every chat, every document, every decision, and let it figure things out.

That sounds great until you actually try to work with it.

The problem is context quality. If an agent is helping me with a Garmin watch face, it does not need the entire history of a legal patent dispute, old investment research, personal presentation notes, and every half-baked blog idea I ever wrote. More context is not automatically better context. Often it is just noise with confidence.

So I started moving toward isolated memory spaces:

• one knowledge base for a project
• one knowledge base for a legal matter
• one knowledge base for a product experiment
• one broader personal universe of notes in Obsidian

The point is not to prevent cross-project learning forever. The point is to make the default context clean.

When an agent enters a project, it should get the memory that matters for that project first.

The memory loop: flush, compile, query

The system I use now has a small vocabulary.

Session → /memflush → knowledge-base/daily/YYYY-MM-DD.md
                ↓
             /compile
                ↓
knowledge-base/concepts/ + knowledge-base/connections/

At the end of a useful session, /memflush extracts the important parts of the conversation into a daily source log. Decisions, lessons, gotchas, action items, and context that future agents should not have to rediscover all go there.

The phrase “useful session” is doing a lot of work. I do not want every interaction with an agent to become memory. Some sessions are false starts. Some are debugging dead ends. Some are me poking around with an idea I will never revisit. Some are just operational noise. If all of that gets stored automatically, the memory bank gets bigger but not smarter.

Then /compile turns those daily logs into more durable knowledge articles:

• concepts/ for atomic topics
• connections/ for relationships between topics
• index.md as the map the agent reads first
• log.md as the history of how the knowledge base evolved

There is also /query, which uses the knowledge base as a targeted retrieval layer. Instead of asking the agent to search my entire life, I ask a project knowledge base what it knows.

That sounds simple because it is simple. The usefulness comes from the discipline of turning chat residue into structured project memory.

Why Markdown won

I keep coming back to Markdown because it has the right failure modes.

If an agent writes a bad note, I can open it. If a summary is too vague, I can edit it. If a project has stale concepts, I can delete or rewrite them. The knowledge base is not hidden inside an embedding index or a proprietary memory layer where I have to trust that the right thing will surface later.

Markdown also makes the system portable across tools. I have used different coding agents and workflows over the year, including OpenCode earlier and now pi-agent because it is more configurable for my setup. The important part is that the memory does not belong to one agent vendor. It belongs to the project.

That separation matters. Agents change fast. Models change fast. Tooling changes fast. Plain text survives.

Why I moved toward pi-agent

The reason pi-agent clicked for this workflow is configurability.

I wanted hooks, skills, project instructions, memory commands, and tool access to behave differently across different kinds of work. A watch-face project needs one context. A company knowledge base needs another. A legal research project needs something stricter. A blog-writing workflow needs frontmatter and publishing conventions.

There is one place where pi-agent initially felt worse than tools like Claude Code: it does not automatically run my memory flush at the end of every session.

At first that annoyed me. If the whole point is memory, why not hook session end and store everything?

I have mostly changed my mind. The missing automatic flush is actually a useful forcing function. It keeps me in control of what deserves to enter the memory system. If a session produced a real lesson, I run /memflush. If it was noise, I let it disappear.

That matters because stale or low-value information is not neutral. Bad memory is worse than no memory. The more junk you store, the harder future agents have to work to separate signal from residue. Humans have the same problem. Sleep is not just storage. It is consolidation, pruning, and housekeeping.

A fully automatic memory hook optimizes for capture. I want to optimize for useful recall.

The more I used agents across different suppliers and interfaces, the more I wanted the memory system to sit underneath them instead of being trapped inside them. Pi made it easier to wire those workflows together without bloating every session with irrelevant context.

That is the key design principle: optimize the context every time.

Do not load everything because you can. Load what helps the current task. Do not store everything because you can. Store what future you will actually want the agent to know.

Project memory beats universal memory most of the time

The biggest practical lesson is that project-local memory usually beats universal memory.

A project knowledge base can be opinionated. It can say:

• here is the architecture
• here are the commands that work
• here are the simulator gotchas
• here are the decisions we already made
• here are the concepts worth remembering
• here is what not to try again

That is much more useful than a generic assistant remembering that I once worked on Garmin watch faces.

For example, my Garmin watch-face project has notes about resolution buckets, MIP display palettes, simulator screenshot problems, store publishing, and KiezelPay integration. Those details are extremely relevant inside that project and mostly irrelevant elsewhere.

My Cackle project has a different knowledge base. My personal research has another. My broader Obsidian vault acts as the wider universe, but project context stays scoped.

The tradeoff is obvious: you lose some effortless cross-project awareness. But in practice the cleaner context is worth it. If I need cross-project synthesis, I can ask for it deliberately. I do not want it leaking into every session by default.

The agent maintains its own memory

The most important shift is that the agent is not just consuming memory. It helps maintain it.

When a session produces useful knowledge, the agent can flush it. When daily logs pile up, it can compile them into concept pages. When concepts get stale, it can update the index. When a project starts, it can initialize a knowledge base from existing files.

That creates a feedback loop:

1. Work happens.
2. The agent extracts what mattered.
3. The knowledge base becomes better.
4. The next agent session starts with cleaner context.
5. Work gets easier.

This is the part that feels qualitatively different from traditional note-taking. I still review and steer, but I am no longer the only person responsible for keeping the project memory usable.

The structure that seems to work

A typical project knowledge base looks like this:

knowledge-base/
├── index.md
├── log.md
├── AGENTS.md
├── daily/
├── concepts/
├── connections/
├── qa/
└── reports/

The daily logs are source material. They are allowed to be chronological and messy. The concepts are the cleaned-up encyclopedia. The connections capture relationships that do not belong to one concept. The index is what the agent reads first.

This separation matters because it prevents every note from trying to do everything. Daily logs preserve evidence. Concepts compress. Connections synthesize. The index orients.

It is a small architecture, but it has held up surprisingly well.

What still hurts

This is not solved. It is a working system with obvious rough edges.

The first problem is discipline. Memory only improves if useful sessions get flushed and compiled. If I skip that step for weeks, the knowledge base gets stale like any other documentation. Manual flushing is tedious, but that tedium is also a filter. It makes me ask: is this actually worth remembering?

The second problem is quality control. Agents are good at summarizing, but they can still overgeneralize, bury important caveats, or create duplicate concepts. That means the knowledge base needs occasional linting and pruning.

The third problem is memory inflation. A system that automatically stores everything will eventually fill itself with low-quality context: abandoned plans, one-off experiments, wrong turns, and stale assumptions. That looks like more knowledge, but behaves like worse context.

The fourth problem is boundaries. Project isolation is useful, but sometimes knowledge really should cross boundaries. The trick is to make cross-project memory explicit instead of accidental.

The fifth problem is trust. A compiled concept can look authoritative even if it was based on a bad daily note. Source links back to daily logs help, but they do not remove the need for judgment.

What I would build next

The next version should make the maintenance loop easier without making it mindless.

I want:

• better reminders to flush after meaningful sessions
• no default “store everything” session-end hook
• optional prompts that ask whether a session deserves memory
• automatic detection of stale concepts
• cleaner cross-project querying when I deliberately ask for it
• better lint reports for contradictions and duplicate ideas
• project dashboards showing what the agent thinks it knows
• visual maps of which memories are active in a session

The last one matters. Context should be inspectable. If an agent is using a memory system, I want to know what it loaded and why.

My scoptimist take

I am optimistic about AI memory, but skeptical of magical memory.

The useful version is not “the AI remembers everything about me.” The useful version is “each project has a small, inspectable knowledge base that gets better as we work.”

That also means memory should be selective. Automatic capture sounds convenient, and I understand why coding agents are adding it. But I do not want to live at the mercy of whatever the agent decides is important. I want a memory system where I can choose when something was worth preserving.

That sounds less impressive in a product demo, but it is much more useful in daily work.

After a year of experiments, my memory system is basically a garden of Markdown files maintained by agents, rooted in Obsidian, scoped by project, and compiled from real sessions. It is not perfect. It is not fully automatic. But it gives agents a place to stand.

And for now, that is the difference between starting every session from scratch and feeling like the work compounds.