The complaint arrives in the same form every time: the agent was great for ten minutes, then it forgot what we were doing.
It did not forget. It never remembered. A context window is not memory — it is a desk. Everything on the desk is available; everything that falls off is gone, and nothing is filed anywhere. Memory is something you build on top.
The four kinds of memory an agent needs
Borrowed from cognitive science, and useful because each one has a different implementation and a different failure mode.
- Working memory — the current task state. What we are doing right now, what has been tried, what came back.
- Episodic memory — what happened before. Previous conversations with this customer, past tickets, prior decisions.
- Semantic memory — what is true about the world. Product facts, policies, definitions. This is your RAG layer.
- Procedural memory — how to do things. The workflows, the escalation rules, the house style.
Most agents ship with only semantic memory, because RAG is the well-known piece. They then feel amnesiac in exactly the way users complain about, because episodic and working memory were never built.
Working memory: the state object
Do not rely on the transcript. Transcripts grow, get truncated, and bury the important parts in conversational filler.
Keep an explicit structured state — the goal, established facts, actions taken, open questions, current hypothesis — and update it as the agent works. Then pass that object between steps rather than the raw history.
Episodic memory: what happened last time
This is what makes an agent feel like it knows you. It needs storage keyed by entity — customer, account, case — and a retrieval step at the start of each interaction that pulls the relevant history in.
The design question is what to keep. Storing every message is expensive and dilutes retrieval. Storing a written summary of each interaction, plus any durable facts learned, works considerably better in practice.
- Summarise each session at the end rather than storing raw logs.
- Extract durable facts separately: preferences, constraints, commitments made.
- Decay or archive the rest. Not every interaction from two years ago deserves to influence today’s answer.
The context window is a budget, not a container
Larger context windows have not removed this problem, for two reasons.
First, cost and latency scale with what you send, and you re-send on every step. Second, models attend unevenly across a long context — burying the critical instruction in the middle of a hundred thousand tokens is a reliable way to have it ignored.
Treat context as a budget you allocate deliberately, not a bucket you fill until it stops erroring.
- System instructions and current task at the top, where attention is strongest.
- Retrieved knowledge in the middle, ranked, few and relevant.
- The immediate exchange at the end.
- Everything else summarised, not included verbatim.
The failure nobody plans for: memory poisoning
Once an agent writes to its own memory, a single wrong conclusion becomes a permanent fact that shapes every future interaction. The customer is recorded as being on the enterprise plan; they are not; every subsequent answer is confidently wrong and self-consistent.
This is the memory equivalent of a data quality problem, and it compounds silently.
- Record where each remembered fact came from and when.
- Separate what the agent was told from what it inferred. Treat inferences as lower confidence.
- Make memory inspectable and editable by a human — you will need to correct it.
- Expire inferred facts. Confirmed ones can persist.
Where to start
If your agent feels forgetful, the order of return is usually: a structured working-state object first, then episodic summaries keyed by entity, then deliberate context budgeting. Semantic memory you have probably already built and called it RAG.
Procedural memory is last and is mostly a prompt and tooling design problem rather than a storage one.
We build agents with memory architectures that hold up past the first ten minutes — including the unglamorous parts like fact provenance and human-editable state. If yours keeps forgetting, that is a design gap rather than a model limitation.