I think this conversation is collapsing storage, retrieval and memory into one thing.
The way I’ve built it, the “brain” has a knowledge layer, three separate memory layers, and a consolidation process that maintains them. None of it belongs to the execution model. The whole system is model agnostic. Claude, GPT, GLM, Qwen, a local model, or whatever comes next can all sit behind the same memory, retrieval and RAG layer. The model is replaceable. The accumulated knowledge and state are not.
The knowledge layer is RAG. My first real corpus is 62,638 chunks with roughly a 1.36 GB vector index. I use Nomic embeddings to turn the source material and the query into vectors, FAISS to find the nearest candidates, date-aware recency weighting, diversification to keep near-duplicate material from flooding the result set, and a Qwen3 reranker to score the remaining candidates against the actual question.
At that scale, file naming and folders are not really the problem anymore. The hard problem is deciding which five or ten pieces of information deserve to reach the model.
Recency matters too. A semantically perfect email from four years ago may be less useful than a nearly identical one from last month. I don’t simply sort newest-first either. Date becomes part of the score with a decay curve. Old information can still win if it is substantially more relevant.
Then there are three actual memory layers.
Working memory is the current conversation and immediate state. What I said 30 seconds ago should have more authority over the current task than something retrieved from six months ago. This also means you need precedence rules so stale summaries or recalled information cannot override the user’s current instruction.
Long-term memory contains facts, decisions, project state, conclusions, relationships, prior outcomes and other information worth recalling later. That gets embedded separately from the document corpus. Recall is not just vector similarity. I score memories using semantic relevance, freshness, novelty, decay and prior access, then inject only a small top slice into context.
Persistent memory is different again. These are things that should behave more like operating instructions than recalled facts. User preferences, stable rules, skills, company policies and similar information should not compete with ordinary memories or disappear simply because they have not been mentioned recently.
Then there is dreaming, which is the consolidation process.
Periodically the system reviews new conversations against existing memory and decides what is worth keeping. New information can be a duplicate, an augmentation of something already known, or a contradiction.
Contradiction handling is a much bigger problem than “store everything.”
If I said in January that a customer’s target was $10M and in June changed it to $12M, a system that perfectly stores both statements still has a memory problem. It needs to know that the second statement supersedes the first. I keep the old fact for history and provenance, but expire it from normal recall and link it to the new one. That means historical truth is preserved without stale truth competing with current truth.
Forgetting matters too. Not every memory should remain equally important forever. Long-term memories can decay. Repeated or reinforced information can strengthen. Old information can remain available without continuing to dominate recall.
Then you get to context hygiene.
Retrieval can search gigabytes. The execution model should see kilobytes.
I intentionally cap recalled memory before it reaches the model. The point is not to prove how much context I can stuff into a context window. The point is to give the model the smallest amount of high-signal context required to answer the current request.
That is also why “stores everything” does not tell me much about a memory system. I care about what it retrieves, how it ranks it, whether it understands time, what happens when facts conflict, what decays, what becomes persistent, what gets excluded, and what is allowed into the current context.
I also keep RAG and memory separate. A 30 GB email archive, SOP library, customer documents, drawings or research corpus are knowledge sources. They are not automatically memories. The system can retrieve from them when needed without pretending the model permanently “knows” all of it.
And because the whole thing sits outside the execution model, I can change models without losing the brain. I can change the embedding model. I can change the reranker. I can change the vector store. I can change the execution model entirely. The accumulated memory, provenance, permissions, conflict history and knowledge remain intact.
So I personally would not start a second-brain design with Obsidian vs Computer vs another vault.
I’d start with the memory lifecycle, retrieval policy, precedence rules and model-independent context layer.
Storage can change. Models can change. The brain should survive both.
To me, that is the actual second brain. It is the layer that knows what you know, what changed, what is still true, what matters now, and which parts a given model should see for this specific task.
