LLMRAGOpen SourceBenchmarkingMemory Architecture

MEMORYLENS: BENCHMARKING LLM MEMORY DECAY ACROSS 100 CONVERSATION TURNS

I built MemoryLens after noticing a glaring gap: every RAG tutorial claims superior recall, but nobody had actually measured how memory architectures degrade as conversations grow longer. So I quantified it.

5.45×

Recall/token improvement

85%

Chunked RAG recall @ T=100

78%

Lower cost vs naive

100

Conversation turns tested

THE PROBLEM WITH LLM MEMORY

LLMs forget. When a conversation exceeds the context window, tokens get dropped — and the model loses facts it was told 30 messages ago. Naive truncation (just dropping old tokens) is the default behaviour in most production chatbots. RAG-based memory retrieves relevant chunks instead of dropping them blindly. But how much better is it, really, at turn 50? Turn 100?

I modelled memory decay using Ebbinghaus forgetting curves alongside exponential and linear decay models, then ran statistical multi-seed benchmarks across five memory backends.

FIVE MEMORY ARCHITECTURES TESTED

1. Naive → Sliding window truncation. Baseline. 2. RAG → Dense retrieval (all-MiniLM-L6-v2) on full history. 3. Chunked RAG → Chunked dense retrieval, top-8 → rerank top-4. 4. Cascading → Temporal decay-weighted retrieval. Best efficiency. 5. SummaryMemory → LLM-generated rolling summaries.

EVALUATION METRICS

All five metrics are computed without requiring an API key — content-based evaluation only:

Recall@T — Is the fact present in retrieved context at turn T? Precision@K — What fraction of retrieved chunks are relevant? Temporal Drift — How much stale/contaminated content leaks through? Memory Noise — Ratio of irrelevant content in the context window. Cascade Efficiency — Recall improvement per token consumed.

BENCHMARK RESULTS

At turn 100, Chunked RAG achieves 85% recall using 38 tokens/query. Cascading Temporal delivers 5.45× better recall-per-token efficiency than naive truncation. Naive truncation bottoms out at 31% recall.

SummaryMemory showed high recall but suffered from hallucination drift — the LLM-generated summaries introduced factual contamination at scale, making it unreliable for high-stakes applications.

WHY THIS MATTERS FOR PRODUCTION RAG

If you're building a customer support chatbot, a coding assistant, or any long-horizon conversational AI and using naive truncation, you're silently losing context that directly impacts response quality. MemoryLens gives you a reproducible framework to test your memory strategy before it fails in production.

OPEN SOURCE

MemoryLens is fully open-source. It ships with 24 integration tests, CI via GitHub Actions (Python 3.10–3.11), a Streamlit dashboard, and supports Groq, OpenAI, Anthropic, OpenRouter, and Ollama as LLM backends — no API key required for the content-based metrics path.

View on GitHub →