Short-term and long-term memory

How it works: short-term memory

1. Short-term memory is the context your loop resends each turn: system prompt, recent messages, tool results, the current plan.
2. It has a budget: the context window, and your cost and latency limits.
3. Keep the most useful parts: the latest turns verbatim; important facts pinned (the user's goal, constraints).
4. Shrink the rest: drop old tool results, summarise older turns (compaction), or move them to long-term memory.
5. Per-session state (a shopping cart, a draft document) is better kept as structured data than as chat text.

How it works: long-term memory

1. Long-term memory lives outside the model: a database, key-value store, vector store or files.
2. Decide what to save:
   • Semantic (facts and preferences): "vegetarian", "prefers metric units", "works in finance".
   • Episodic (events): "On 3 Oct we fixed the login bug by resetting SSO."
   • Procedural (learned how-tos): "For this repo, run make test, not pytest."
3. Save it: through a memory tool the agent calls ("remember that…"), a background job that extracts facts after each session, or explicit user settings.
4. Retrieve it: at the start of a session, or when relevant to the current request, by key (user ID) or by similarity search (like RAG).
5. Inject only what's relevant into the context, labelled as memory.
6. Govern it: users can view and delete memories; apply retention limits; never store secrets; keep memories per user (and per tenant) with access control.
7. Handle conflicts and staleness: newer facts replace older ones ("moved to Berlin" replaces "lives in London").

from collections import deque
 
# SHORT-TERM MEMORY: the recent conversation, trimmed to a budget.
class ShortTerm:
    def __init__(self, max_messages=4):
        self.messages = deque(maxlen=max_messages)   # oldest messages fall off automatically
    def add(self, role, text):
        self.messages.append({"role": role, "content": text})
 
# LONG-TERM MEMORY: facts saved across sessions, looked up when relevant.
class LongTerm:
    def __init__(self):
        self.facts = {}                              # a real app: a database or vector store
    def remember(self, user, key, value):
        self.facts.setdefault(user, {})[key] = value
    def recall(self, user, question):
        return {k: v for k, v in self.facts.get(user, {}).items() if k in question.lower()}
 
short, long_term = ShortTerm(), LongTerm()
for i, text in enumerate(["Hi", "I'm vegetarian", "Book a table", "For 4 people", "At 7pm"]):
    short.add("user", text)
long_term.remember("u42", "diet", "vegetarian")      # saved by a memory tool or a background job
 
print("short-term window:", [m["content"] for m in short.messages])
print("long-term recall :", long_term.recall("u42", "Suggest a diet-friendly restaurant"))

short-term window: ["I'm vegetarian", 'Book a table', 'For 4 people', 'At 7pm']
long-term recall : {'diet': 'vegetarian'}