LLMRAGOpen SourceBenchmarkingMemory Architecture

MEMORYLENS: BENCHMARKING LLM MEMORY DECAY ACROSS 100 CONVERSATION TURNS

I built MemoryLens after noticing a glaring gap: every RAG tutorial claims superior recall, but nobody had actually measured how memory architectures degrade as conversations grow longer. So I quantified it.

5.45×

Recall/token improvement

85%

Chunked RAG recall @ T=100

78%

Lower cost vs naive

100

Conversation turns tested

THE PROBLEM WITH LLM MEMORY

LLMs forget. When a conversation exceeds the context window, tokens get dropped, and the model loses facts it was told 30 messages ago. Naive truncation (just dropping old tokens) is the default behaviour in most production chatbots. RAG-based memory retrieves relevant chunks instead of dropping them blindly. But how much better is it, really, at turn 50? Turn 100?

I modelled memory decay using Ebbinghaus forgetting curves alongside exponential and linear decay models, then ran statistical multi-seed benchmarks across five memory backends.

FIVE MEMORY ARCHITECTURES TESTED

1. Naive → Sliding window truncation. Baseline. 2. RAG → Dense retrieval (all-MiniLM-L6-v2) on full history. 3. Chunked RAG → Chunked dense retrieval, top-8 → rerank top-4. 4. Cascading → Temporal decay-weighted retrieval. Best efficiency. 5. SummaryMemory → LLM-generated rolling summaries.

EVALUATION METRICS

All five metrics are computed without requiring an API key. Content-based evaluation only:

Recall@T : Is the fact present in retrieved context at turn T? Precision@K : What fraction of retrieved chunks are relevant? Temporal Drift : How much stale/contaminated content leaks through? Memory Noise : Ratio of irrelevant content in the context window. Cascade Efficiency : Recall improvement per token consumed.

BENCHMARK RESULTS

At turn 100, Chunked RAG achieves 85% recall using 38 tokens/query. Cascading Temporal delivers 5.45× better recall-per-token efficiency than naive truncation. Naive truncation bottoms out at 31% recall.

SummaryMemory showed high recall but suffered from hallucination drift: the LLM-generated summaries introduced factual contamination at scale, making it unreliable for high-stakes applications.

WHY THIS MATTERS FOR PRODUCTION RAG

If you're building a customer support chatbot, a coding assistant, or any long-horizon conversational AI and using naive truncation, you're silently losing context that directly impacts response quality. MemoryLens gives you a reproducible framework to test your memory strategy before it fails in production.

OPEN SOURCE

MemoryLens is fully open-source. It ships with 24 integration tests, CI via GitHub Actions (Python 3.10–3.11), a Streamlit dashboard, and supports Groq, OpenAI, Anthropic, OpenRouter, and Ollama as LLM backends, no API key required for the content-based metrics path.

View on GitHub →